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ABSTRACT 


The  importance  of  reliable  and  maintainable  equipment 
in  the  hands  of  our  service  members  cannot  be  overstated. 
Reliability  has  been  identified  as  the  life  cycle  cost 
driver  for  defense  weapon  systems.  Knowing  that  the 
reliability  of  a  weapon  system  directly  impacts  upon  that 
system' s  operational  capability  and  life  cycle  costs  makes 

it  of  fundamental  importance  to  the  warfighter.  In 

recognition  of  its  importance,  it  is  mandatory  for  all 

program  managers  within  the  Department  of  Defense  (DoD)  to 
ensure  their  program  accounts  for  the  user' s  reliability 
objectives.  However,  reliability  failures  continue  to 

disappoint  operators,  maintainers  and  testers  of  DoD 

systems . 

This  thesis  evaluates  reliability  management  within 
the  acquisition  process  of  Naval  Aviation  programs. 
Reliability,  logistical,  and  program  management  personnel 
directly  involved  with  the  issues  of  reliability  management 
provided  empirical  insight  to  help  the  researcher  identify 
root  causes  and  risk  mitigation  techniques  that  are 
critical  to  reliability  optimization. 
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I. 


INTRODUCTION 


A.  PURPOSE 

The  purpose  of  this  thesis  is  to  examine  the 
managerial  methods  and  practices  used  to  optimize 
reliability  of  weapon  systems  in  selected  Naval  Aviation 
acquisition  programs.  Based  on  the  aforementioned,  a 
secondary  purpose  is  to  examine  opportunities  for 
improvement.  The  objective  of  this  endeavor  is  to 
determine  how  reliability  management  can  best  serve  the 
ultimate  acquisition  goal  of  equipping  warfighters  with  the 
tools  they  required  while  maximizing  available  resources. 
This  analysis  may  lead  to  recommendations  for 
implementation  by  acquisition  workforce  personnel. 

Examination  of  multiple  aspects  of  reliability 
management  will  identify  program  risks  and  opportunities  to 
mitigate  those  risks.  Ultimately  the  goal  is  to  identify 
reliability  management  issues,  which  may  affect  reliability 
optimization  throughout  the  stages  of  the  acquisition  life 
cycle . 


B .  BACKGROUND 

The  1998  National  Research  Council  study  of 
Statistics,  Testing  and  Defense  Acquisition  recommended 
that,  "The  Department  of  Defense  and  the  military  services 
should  give  increased  attention  to  their  reliability, 
availability,  and  maintainability  data  collection  and 
analysis  procedures  because  deficiencies  continue  to  be 
responsible  for  many  of  the  current  field  problems  and 
concerns  about  military  readiness."  [Ref.  2]  In  the  period 
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between  1996  and  2000,  approximately  80  percent  of  Army 
systems  tested  failed  in  excess  of  half  of  their 
reliability  requirements  during  operational  testing.  [Ref. 
12] 

Reliability  is  a  fundamental  component  of  operational 
capabilities  of  weapon  system,  as  well  as  a  primary  cost 
driver  of  that  system' s  life  cycle  costs  (LCC) .  Either  of 
these  issues  is  expected  to  be  of  the  utmost  concern  to  our 
nation's  warfighters.  The  combination  of  operational 
capability  and  LCC  makes  the  issue  of  weapon  system 
reliability,  a  paramount  concern  for  every  acquisition 
professional.  The  Director,  Operational  Test  and 
Evaluation  has  taken  renewed  efforts  to  improve  the  process 
of  reliability  testing  in  hopes  of  fielding  operationally 
suitable  systems  to  the  end  user.  [Ref.  13] 

However,  testing  is  only  one  tool  available  to  a 
program.  Multiple  opportunities  are  present  throughout  the 
acquisition  life  cycle  to  address  reliability.  Beginning 
with  the  initial  requirements  development,  through  each 
iteration  of  the  systems  engineering  process,  and 
ultimately  during  post-production  support  reliability  must 
be  planned  for,  monitored,  accessed,  and  improved  during 
the  maturation  of  a  weapon  system. 

C .  RESEARCH  QUESTIONS 

The  primary  research  question  is: 

What  strategies  should  Program  Managers  implement  to 
optimize  reliability  in  their  weapon  system? 
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The  subsidiary  research  questions  are  as  follows: 

1 .  What  is  reliability  and  what  is  its  significance 
within  acquisition  management? 

2 .  What  policies  and  regulations  governing 

reliability  management  are  available  to  Program 
Managers  (PM) ? 

3.  What  significant  factors  contribute  to  weapon 
system  reliability? 

4.  What  strategies  are  currently  used  to  monitor 
reliability  within  an  acquisition  program? 

5.  How  can  the  program  office  mitigate  risk 
associated  with  reliability  throughout  the 
acquisition  life  cycle? 


D .  SCOPE 

This  thesis  examined  reliability  management  from  a 
program  management  perspective  throughout  the  acquisition 
life  cycle.  Governing  policies  and  regulatory  documents 
have  been  reviewed  from  the  viewpoint  of  a  PM  with  respect 
to  applicability  as  a  managerial  tool.  The  focus  of  the 
data  collected  will  be  from  current  systems  in  various 
stages  of  development  managed  at  the  Naval  Air  Systems 
Command  (NAVAIR) ,  and  limited  to  managerial  processes, 
tools  and  techniques.  This  research  is  intended  to 
identify  both  enablers  and  inhibitors  of  effective 
reliability  management.  The  resulting  analysis  will  aide 
in  the  development  of  a  set  of  conclusions  and 
recommendations  applicable  to  acquisition  professionals  in 
current  and  future  endeavors . 
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E .  METHODOLOGY 

This  thesis  sought  to  determine  the  current 
environment  for  reliability  management  within  Naval  Air 
acquisition.  This  task  was  accomplished  through  a  thorough 
examination  of  literature,  reports,  and  regulatory 
documents.  Additionally,  the  researcher  obtained 

information  through  the  conduct  of  personal  and  telephonic 
interviews  with  acquisition  professionals.  The  final  method 
of  data  collection  was  an  electronic  survey  of  specific 
acquisition  programs  managed  at  NAVAIR.  The  survey  used 
was  a  modification  of  a  previously  designed  reliability 
performance  survey  intended  to  gather  data  from  within  a 
specific  Army  Program  Executive  Office  in  pursuit  of 
similar  research  objectives.  [Ref.  39] 

The  information  collected  during  the  conduct  of  the 
reliability  performance  survey,  was  obtained  through 
Government  and  contractor  personnel  who  had  first-hand 
experience  with  reliability  management  and/or  reliability 
engineering.  Responses  to  the  survey  were  instrumental  in 
construction  of  this  thesis.  Respondents  provided  a  good 
cross-section  of  programs  in  various  stages  of  the 
acquisition  life  cycle.  The  questions  posed  were  intended 
to  emphasize  the  perspective  of  program  management  on  the 
varied  tasks  involved  with  reliability  management. 

F.  ORGANIZATION 

This  thesis  contains  five  chapters. 

Chapter  I  provides  an  introduction  to  the  subject  of 
reliability  and  a  basis  for  the  study,  outlining  the  scope. 
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methodology,  and  structure  for  conducting  the  analysis 
which  will  be  used  to  address  the  research  questions. 

Chapter  II  will  provide  a  foundation  of  information  on 
reliability,  regulatory  documents  and  opportunities  to 
affect  reliability  throughout  the  acquisition  life  cycle. 

Chapter  III  will  present  the  information  obtained  from 
the  research  conducted  about  the  NAVAIR  programs  and  the 
results  of  the  reliability  performance  survey.  This  data 
will  indicate  how  the  various  programs  have  implemented 
reliability  management  processes  and  practices  and  will 
highlight  significant  examples  and  experiences. 

Chapter  IV  will  provide  the  analysis  of  the  collected 
data  and  identify  the  techniques  and  strategies  used  to 
manage  reliability  within  the  surveyed  acquisition 
programs . 

Chapter  V  will  present  findings,  address  the  primary 
and  subsidiary  research  questions,  and  make  conclusions  and 
recommendations . 

G.  BENEFITS  OF  STUDY 

The  intended  beneficiaries  of  this  research  are  the 
warfighters  within  our  military  services,  through  the 
acquisition  programs  that  manage  reliability  within  their 
weapon  systems.  The  compilation  of  lessons  learned  and  the 
identification  of  processes,  policies  and  strategies 
pertaining  to  reliability  management  can  assist  any  program 
attempting  to  apply  risk  management  with  respect  to 
reliability  performance. 
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The  examination  of  common  practices,  issues,  and 
concerns  of  programs  wrestling  with  reliability  issues  will 
provide  a  stepping  stone  for  improvement  or  root  cause 
determination  for  future  programs  allowing  them  to 
capitalize  on  the  lessons  of  their  predecessors. 
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II.  RELIABILITY  OVERVIEW 


Reliability  is  the  single  most  powerful  lever  in 
all  logistics-if  it  doesn't  break  it  doesn't  need 
support  (Ref.  21) 

A.  INTRODUCTION 

"Reliability  isn't  everything,  it  is  the  only  thing." 
[Ref.  17]  The  nature  of  National  Defense  and  our  system  of 
Government,  has  led  to  a  complex  systems  acquisition  system 
that  produces  equally  complex  and  expensive  weapon  systems. 
The  costs  of  the  weapon  systems  combined  with  the  Nation' s 
requirement  for  defense  dictate  the  need  for  highly 
reliable  tools  placed  in  the  hands  of  trained  warfighters. 
After  spending  in  excess  of  40  million  in  procurement 

dollars  for  an  aircraft  it  simply  is  not  feasible  to 

discard  the  system  when  it  does  not  work  perfectly. 

Therefore,  it  is  imperative  to  properly  design  and 
manufacture  each  system  to  optimize  its  reliability 

potential . 

This  purpose  of  this  chapter  is  to  establish  a  common 
ground  and  set  the  stage  for  the  discussion  to  follow  about 
reliability  management.  This  will  be  accomplished  through 
a  presentation  of  background  information  intended  to 
provide  a  fundamental  understanding  of  reliability  and  its 
importance  within  systems  acquisition.  Definitions  of 
terms  relating  to  reliability  are  provided  to  facilitate  in 
understanding  and  consistency. 

Mandatory  and  discretionary  documentation  have  been 
identified  and  discussed.  Additionally,  the  discussion 

focuses  on  the  importance  of  reliability  management  within 
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systems  acquisition  and  opportunities  to  optimize 
reliability  will  be  highlighted 

B.  DEFINITIONS  AND  CONCEPTS 

Surprisingly,  definitions,  as  well  as  the  terms 
themselves,  seem  to  vary  depending  on  the  user,  service 
component,  specific  system  program,  or  source 
documentation.  This  seems  to  be  unexpected  knowing  that 
the  discipline  of  reliability  falls  in  the  realm  of 
engineering,  which  tends  to  maintain  uniform  definitions 
even  while  measurement  methods  vary.  [Ref.  7]  Prior  to  the 
1996  specification  and  standards  reform,  MIL-STD-721 
"Definition  of  Terms  for  Reliability  and  Maintainability" 
appropriately  stood  as  a  definitive  authority.  After  the 
cancellation  of  this  standard  several  'authoritative' 
sources  are  often  quoted. 

The  IEEE  Reliability  Society  has  yet  to  publish  a 
commercial  standard.  Mil-STD-785B  "Reliability  Program  for 
Systems  and  Equipment  Development  and  Production"  remains  a 
common  reference,  and  portions  of  it  often  are  included  in 
the  Statement  of  Work  (SOW) .  Rome  Laboratories'  The 
Reliability  Toolkit  is  an  excellent  reference,  and  is 
currently  considered  a  commercial  standard  for  practicing 
reliability  engineers.  Finally,  the  Defense  Systems 
Management  College  (DSMC)  refers  to  the  Reliability , 
Availability ,  and  Maintainability  (RAM)  Dictionary ,  by 
Tracy  Omdahl  the  Webster's  of  RAM  terms.  [Ref.  7] 

There  are  more  than  2000  terms  defined  in 
documents  reviewed  so  far,  many  of  which  have  the 
same  meaning  but  different  definitions.  (Ref. 

40) 
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It  is  therefore,  important  to  have  a  solid 
understanding  of  terms  and  their  usage,  relative  to  a 
particular  project.  The  following  common  terms  and 
concepts  are  provided  to  give  the  reader  an  understanding 
of  terminology  widely  used  within  the  practice  of 
reliability . 

Reliability,  Availability,  and  Maintainability 
(RAM) .  Requirement  imposed  on  acquisition  systems  to 
insure  they  are  operationally  ready  for  use  when  needed, 
will  successfully  perform  assigned  functions,  and  can  be 
economically  operated  and  maintained  within  the  scope  of 
logistics  concepts  and  policies.  [Ref.  10] 

Reliability.  The  probability  an  item  will 
perform  its  intended  functions  for  a  specified  period  under 
stated  conditions.  Informally:  It  does  what  it  is 
supposed  to,  for  as  long  as  you  need  it.  Commonly 
quantified  in  terms  of  mean  time  between  failure  (MTBF) . 

Mission  Reliability.  The  ability  of  an  item  to 
perform  its  required  functions  for  the  duration  of  a 
specified  period.  Informally:  A  measure  of  system's 
ability  to  complete  its  mission. 

Logistics  Reliability.  The  probability  that  no 
corrective  maintenance  or  unscheduled  supply  demand  will 
occur  following  the  completion  of  a  specific  mission 
profile.  Informally:  A  measure  of  system's  ability  to 
operate  without  logistics  support. 
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Logistics  Reliability 

Mission  Reliability 

•  Measure  of  system' s 

ability  to  operate 
without  logistics 
support 

•  Measure  of  system' s 

ability  to  complete 
mission 

•  Recognize  effects  of  all 

occurrences  that  demand 
support  without  regard 
to  effect  on  mission 

•  Consider  only  failures 

that  cause  mission 
abort 

•  Degraded  by  redundancy 

•  Improved  by  redundancy 

•  Usually  equal  to  or 

lower  than  mission 
reliability 

•  Usually  higher  than 

logistics  reliability 

Table  1.  Reliability  Requirements  and  Characteristics 

[From  Ref.  38] 

Maintainability .  The  probability  that  an  item 


will  conform  to  specified  conditions  within  a  given  a  given 
period  when  corrective  or  preventative  action  is  performed 
lAW  prescribed  procedures  and  resources.  Informally:  It 
is  quick  and  easy  to  fix  when  it  breaks. 

Ava liability.  A  measure  of  the  degree  to  which 
an  item  is  in  operable  and  committable  state  at  the  start 
of  a  mission  when  the  mission  is  called  for  at  an  unknown 
(random)  time.  Informally:  It  is  there  and  working  when 
you  call  upon  it. 

Operational  Availability  (Aq)  .  The  probability 
that  an  item,  when  used  under  stated  conditions  in  an 
actual  operational  environment,  will  operate  satisfactorily 
when  called  upon.  Informally:  It  is  mission  ready  when  I 
need  it  at  any  random  time. 

Operational  availability  is  a  preferred  readiness 
measure  for  weapon  systems,  because  it  reflects  the  real- 
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world  operating  environment.  [Ref.  37]  Commonly 
operational  availability  is  considered:  Ao= (Uptime) / (Uptime 
+  Downtime) .  Downtime  consists  of  both  scheduled  and 
unscheduled  maintenance  and  any  administrative  delays  (e.g. 
time  spent  waiting  for  parts  not  in  stock) .  Therefore,  Aq 
is  what  system  operators'  care  most  about--Perf ormance  on 
demand . 

Mean  Time  Between  Failure  (MTBF)  .  For  a 
particular  interval,  the  total  functional  life  of  a 
population  of  an  item  divided  by  the  total  number  of 
failures  within  the  population.  MTBF  is  the  basic 
technical  measure  of  reliability.  [Ref.  10] 

Life  Cycle  Cost  (LCC)  .  The  total  cost  to  the 
Government  of  acquisition  and  ownership  of  that  system  over 
its  useful  life.  It  includes  the  cost  of  development, 
acquisition,  operations,  and  support  (to  include  manpower) , 
and  where  applicable,  disposal. 


C.  POLICY  AND  REGULATIONS 

1.  DoD  5000. 2-R,  Mandatory  Procedures  for  Major 
Defense  Acquisition  Programs  (MDAPS)  and  Major 
Automated  Information  System  (MAIS)  Acquisition 
Programs 

This  regulation  sets  mandatory  procedures  to  be 
followed  by  PMs  and  has  specific  identified  reliability 
requirements : 

The  PM  shall  establish  RAM  activities  early  in 
the  acquisition  cycle.  The  PM  shall  develop  RAM 
system  requirements  based  on  the  ORD  and  TOC 
considerations,  and  state  them  in  quantifiable, 
operational  terms,  measurable  during  DT&E  and 
OT&E.  RAM  system  requirements  shall  address  all 
elements  of  the  system,  including  support  and 
training  equipment.  They  shall  be  derived  from, 
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and  support,  the  user's  system  readiness 
objectives.  Reliability  requirements  shall 
address  mission  reliability  and  logistic 
reliability.  Availability  requirements  shall 
address  the  readiness  of  the  system. 
Maintainability  requirements  shall  address 
servicing,  preventive,  and  corrective 
maintenance . 

The  PM  shall  plan  and  execute  RAM  design, 
manufacturing  development,  and  test  activities  so 
that  the  system  elements,  including  software, 
used  to  demonstrate  system  performance  before  the 
production  decision  reflect  the  mature  design. 
lOT&E  shall  use  production  representative 
systems,  actual  operational  procedures,  and 
personnel  with  representative  skill  levels.  To 
reduce  testing  costs,  the  PM  shall  utilize  M&S  in 
the  demonstration  of  RAM  requirements  wherever 
appropriate . 

This  policy  applies  not  only  to  the  system,  but 
also  to  technical  manuals,  spare  parts,  tools, 
and  support  equipment.  [Ref.  14] 


2.  DoD  5000.1,  The  Defense  Acquisition  System 


This  directive  describes  management  principles 
applicable  to  Defense  acquisition  programs,  provides 
mandatory  policies  and  procedures  for  the  management  of 
acquisition  programs,  except  when  statutory  requirements 
override . 

Logistics  transformation  shall  be  accomplished 
through...  A  support  environment  that  maintains 
long-term  competitive  pressures;  continuous 
improvement  of  weapon  system  reliability, 
maintainability,  and  supportability  through 
technology  refreshment  and  other  means;  and 
effective  integration  of  weapon  system-focused 
support  to  provide  total  mission  logistics  and 
optimum  support  to  the  user.  Acquisition  program 
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managers  shall  focus  on  logistics  considerations 
early  in  the  design  process  to  ensure  that  they 
deliver  reliable  systems  that  can  be  cost- 
effectively  supported  and  provide  users  with  the 
necessary  support  infrastructure  to  meet 
peacetime  and  wartime  readiness  requirements. 
[Ref.  15] 


3.  Military  Standards  and  Handbooks 

In  June  1994,  the  Secretary  of  Defense,  Dr,  William 
Perry,  issued  a  memorandum  entitled  Specifications  and 
Standards— A  New  Way  of  Doing  Business.  This  acquisition 
reform  memorandum  was  aimed  at  the  acquisition  management 
community  in  an  attempt  to  increase  access  to  commercial 
state-of-the-art  technology  and  to  facilitate  adoption  of 
commercial  best  practices.  Specifically,  the  Secretary 
made  an  immediate  policy  change  requiring  programs  to  use 
performance  specifications  and  non-Government  standards. 
In  cases  where  PMs  determine  that  exact  design  solutions 
are  required,  and  no  suitable  performance  or  commercial 
standard  exists,  they  may  use  military  standards  only  as  a 
last  resort  and  with  the  specific  waiver  approved  by  the 
Milestone  Decision  Authority  (MDA) .  [Ref.  34] 

Use  of  specifications  and  standards  became  available 
primarily  as  guidance  to  Program  Managers.  It  appears  this 
was  a  significant  shift  in  procedure.  The  use  of  military 
standards  had  been  a  fundamental  component  of  reliability 
engineering  and  management. 

The  following  is  a  list  of  frequently  referenced 
military  standards  and  handbooks  (Mil-Stds/Hdbks)  that. 
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although  they  have  been  cancelled, 
available  reference  on  reliability: 


still  remain  an 


•  MIL-HDBK-7 8  ID  Reliability  Test  Methods,  Plans, 
and  Environments  for  Engineering,  Development 
Qualification,  and  Production 

This  handbook  details  techniques  that  may  be 
used  in  reliability  test  programs  but  is  for 
guidance  only.  This  handbook  cannot  be  cited 
as  a  requirement.  [Ref.  25] 

•  MIL-STD-785B  Reliability  Program  for  Systems  and 
Equipment  Development  and  Production 

This  military  standard  contains  reliability 
tasks,  requirements  and  rationale,  which  can  be 
tailored  to  fit  specific  program  needs.  [Ref. 

26] 

•  MIL-STD-1629A  PROCEDURES  EOR  PEREORMING  A  EAILURE 
MODE,  EFFECTS  AND  CRITICALITY  ANALYSIS (FMECA) 


4 .  Other  Discretionary  References 

There  are  multiple  discretionary  references  available 
to  PMs  depending  on  service  and  program  specifics.  Listed 
below  are  discretionary  references  that  have  application 
across  the  spectrum  of  DoD  acquisition  programs: 


a.  Defense  System  Management  College  (DSMC) 
Series  of  Technical  Management  Educational 
Guides . 

A  family  of  guides  written  from  a  non-service 
specific  perspective  published  by  DSMC  intended  for  use  as 
instructional  aides  and  as  practical  references  to  be  used 
by  acquisition  professionals.  In  both  cases  they  are 
written  for  acquisition  management  personnel  who  are 
familiar  with  basic  terms  and  definitions.  Used  as  a  desk 
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reference,  these  manuals  provide  assistance  to  Government 
and  industry  personnel  in  the  execution  of  their  management 
responsibilities.  The  following  guides  have  application  in 
the  management  of  reliability  programs: 

1.  Acquisition  Logistics  Guide  (December  1997). 

2.  Systems  Engineering  Fundamentals  (January  2001). 

3.  Defense  Manufacturing  Management  Guide  for  Program 
Managers  (April  1989) . 

4.  Test  and  Evaluation  Management  Guide  (November 
2001)  . 

b.  Rome  Laboratory  Toolkits 

The  Rome  Research  Site,  now  part  of  a 
consolidated  Air  Force  Research  Laboratory  system,  was 
formerly  known  as  Rome  Laboratory.  Their  Systems 

Reliability  Division  developed  and  published  a  toolkit 
intended  for  use  by  reliability  and  maintainability 
engineers.  The  Reliability  Engineer's  Toolkit  (April  1993) 
is  a  widely  used  technical  reference  document  as  "an 
application  oriented  guide." 

In  1995  the  Rome  Laboratory  released  an  updated 
edition  "Reliability  Toolkit"  Commercial  Practices .  The 
latest  edition  takes  a  broader  view  of  reliability 
engineering  and  management  and  addresses  both  commercial 
and  military  needs.  [Ref.  37] 

5 .  Navy  References 

a.  SECNAVINST  5000.2B,  Implementation  of 
Mandatory  Procedures  for  Major  and  Nonmajor 
Defense  Acquisition  Programs  and  Major  and 
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Nonmajor  Information  Technology  Acquisition 
Programs 

This  document  supports  the  DoD  5000. 2-R 
instruction  on  reliability  management.  Specifically  RAM  is 
identified  as  an  integral  part  of  systems  engineering 
process,  with  emphasis  placed  on  mission  needs  and  reducing 
life  cycle  costs.  Some  direction  is  provided  in  reference 
to  reliability  analysis. 

b.  NAVAIR  INSTRUCTION  4200.36B,  Acquisition 
Plans 

This  document  provides  policy  and  guidance  on  the 
construction  of  Acquisition  Plans  within  NAVAIR. 
Reliability  concerns  are  to  be  addressed  by  logistic 
considerations  with  reference  to  tailoring  and  quality 
assurance  requirements. 


D.  IMPORTANCE  OF  RELIABILITY,  AVAILABILITY  AND 

MAINTAINABILITY  (RAM) 

Reliability,  maintainability  and  availability  of  a 
weapon  system  directly  impact  upon  that  system' s 
operational  capability  and  life  cycle  costs,  thereby  making 
it  of  fundamental  importance  to  the  warfighter.  Because  of 
its  recognized  importance,  it  is  mandatory  for  all  program 
managers  within  the  Department  of  Defense  to  plan  for  and 
execute  measures  to  ensure  their  program  accounts  for  the 
user's  RAM  objectives.  [Ref.  14] 

The  importance  of  reliable  and  maintainable  equipment 
in  the  hands  of  military  service  members  cannot  be 
overstated.  Ultimately  each  system  maturated  through  the 
DoD  acquisition  system,  is  statutorily  required  to  be 
tested  and  evaluated  to  determine  its  operational 
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effectiveness  and  operational  suitability.  Operational 
suitability  most  directly  encompasses  reliability  and 
logistical  considerations.  Informally  a  suitable  weapon 
system  has  been  defined  as 

A  system  that  is  available  for  combat  when 
needed,  is  reliable  enough  to  accomplish  its 
mission,  operates  satisfactorily  with  service 
personnel  and  other  systems,  and  does  not  impose 
an  undue  logistics  burden  in  peacetime  or 
wartime.  [Ref.  2] 

Essentially  effectiveness  equates  to  "Does  it  work?" 
while  suitability  equates  to:  "Can  we  use,  maintain,  and 
support  it?"  Suitability  notably  impacts  the  fielded 
weapon  system;  particularly  in  terms  of  life  cycle  costs. 
It  is  widely  known,  within  the  acquisition  community,  that 
the  costs  of  operating  and  supporting  (O&S)  a  weapon  system 
far  exceed  the  actual  procurement  costs  incurred  through 
the  design,  development  and  production  of  new  systems. 
Figure  1  depicts  a  historical  breakdown  of  life  cycle  costs 
for  a  typical  major  defense  system.  [Ref.  37] 


ENCmEERMG/ 
MANUFACTUfUNC 
DCVELOFMCNT  PHASE 


Figure  1.  Life  Cycle  Cost  Breakdown  of  a  Typical  Weapon 

System  [From  Ref.  31] 
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The  life  cycle  of  a  weapon  system,  which  began  with 
the  initial  determination  of  its  need,  continues  through 
its  design,  development,  production,  deployment  and 
concludes  with  the  system's  disposal.  As  shown  in  Figure 
1,  the  cost  estimation  categories  are  normally  separated 
into  four  overlapping  areas : 

•  Research  and  Development.  Includes  costs  of 

studies;  analysis;  design;  test  and  evaluation; 
pre-production  article  development;  and 

documentation 

•  Investment .  Accounts  for  all  production  and 
deployment  costs  including  any  required  training 
and  military  construction. 

•  Operations  and  Support.  Consists  of  all  cost 
associated  with  usage  and  maintenance  of  the 
fielded  equipment. 

•  Disposal .  Although  a  relatively  small  percentage 
of  LCC,  and  therefore  often  overlooked,  this  area 
accounts  for  demilitarization,  destruction,  or 
deactivation  of  the  system.  [Ref.  31] 

Many  factors  are  involved  with  the  estimation  of  life 
cycle  costs.  Reliability  considerations,  estimates,  and 
the  accuracy  of  those  estimates,  play  a  significant  role 
within  each  of  the  four  overlapping  cost  estimation 
categories.  The  fundamental  objective  of  LCC  reduction 
analysis  is  to  identify  the  cost  drivers  that  most 
significantly  contribute  to  LCC.  This  allows  for  trade  off 
considerations  with  respect  to  different  courses  of  action. 

Although  reliability,  availability,  and 

maintainability  (RAM)  are  inextricably  linked,  even  often 
thought  of  as  a  single  issue,  this  research  will  primarily 
focus  reliability.  This  is  because  reliability  has  been 
determined  to  be  the  significant  driver  of  0  &  S  costs. 

[Ref.  2]  Each  of  these  three  fundamental  logistics 
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attributes  is  critical,  but  both  maintainability  and 
availability  are  directly  affected  by  reliability. 

Reliability  is  the  fundamental  building  block  that 
either  supports  or  hinders  maintainability  and 
availability.  Maintainability  is  focused  on  what  happens 
when  an  item  does  break  (unscheduled  maintenance)  and  on 
the  level  of  support  required  in  preventing  a  system' s 
failure  (scheduled  maintenance) .  Simply  stated,  in  most 
cases  the  more  reliable  an  item  is,  the  less  maintenance 
attention  required.  Availability,  as  discussed  previously, 
benefits  from  any  reduction  of  time  an  item  spends  in  a 
'non-mission'  capable  status  (e.g.  downtime).  Optimized 
reliability,  assuming  no  significant  corresponding  rise  in 
maintainability,  will  contribute  to  the  optimization  of 
a  va  i  1  ab  i  1  i  t  y . 

Currently  the  budget  of  the  Defense  Department  is 
overwhelming  consumed  by  high  costs  of  O&S.  Knowing  that 
60%  to  85%  of  LCC  is  consumed  in  O&S  expenditures,  there 
has  understandably  been  considerable  effort  to  identify 
ways  to  reduce  those  costs.  The  move  to  increase  funds 
available  for  recapitalization  and  modernization  of  legacy 
systems,  will  be  paid  for  substantially  through  the 
reduction  in  O&S  costs.  Figure  2  depicts  a  Navy  initiative 
to  target  methods  to  reduce  O&S  costs. 


19 


Emphasis  On  Reducinq^Q&S.Cost  of  Naval  Forces 


deduce  O&dJ  C^: 


Figure  2 


Reducing  O&S  Costs  [From  Ref.  35] 


Reliability  investments  and  initiatives  are  clearly  on 
the  critical  path  in  the  Navy's  O&S  reduction  plan. 
Reliability  has  been  identified  as  a  cost  driver  of  O&S 
costs.  Figure  3  identifies  the  cost  drivers  the  Navy 
intends  to  address  in  an  effort  to  reduce  O&S  by  30 
percent.  Reliability  improvements  have  been  identified  as 
an  essential  element  in  the  effort  to  reduce  O&S  costs  on 
material  inventory  items.  [Ref.  35] 
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Integrated  Maintenance  /  Supply  System 
Performance  Contracts 


Significant  Totai  Ownership 
Cost  Reduction 


Figure  3.  Reliability  as  a  Cost  Driver  [From  Ref.  35] 


E.  OPPORTUNITIES  AND  TOOLS  FOR  RELIABILITY  MANAGEMENT 

Effective  reliability  management  is  achieved  through 
the  disciplined  system's  engineering  approach.  Systems 
engineering  is  an  interdisciplinary  approach  to  solve  the 
life  cycle  technical  problems  within  an  Integrated  Product 
Process  Development  (IPPD)  framework.  [Ref.  22] 

Systems  engineering  is  the  management  function 
which  controls  the  total  system  development 
effort  for  the  purpose  of  achieving  an  optimum 
balance  of  all  system  elements.  It  is  a  process 
which  transforms  an  operational  need  into  a 
description  of  systems  parameters  and  integrates 
those  parameters  to  optimize  the  overall  system 
effectiveness  [Ref.  9] 
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Combining  the  structured  iterative  format  of  systems 


engineering  with  the  inter-disciplinary 
with  the  IPPD  system,  provides  a 
opportunity  to  employ  optimum  design, 
support  practices.  {Ref.  9] 


teaming  inherent 
program  with  an 
manufacturing  and 
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Figure  4.  R&M  Fundamentals  Timeline  [From  Ref.  30] 


Opportunities  to  influence  reliability  occur 
throughout  a  system's  life  cycle.  Figure  4  depicts  some  of 
the  reliability  fundamental  tools,  techniques  and  documents 
from  an  acquisition  life  cycle  timeline.  Each  stage  is  a 
building  block  for  the  eventual  fielding  and  operation  of 
the  equipment  by  the  warfighter.  Like  SEP  itself, 
reliability  management  is  an  iterative  task  requiring 
constant  attention  and  optimization  of  trade-offs. 
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1.  Conceptualization,  Design  and  Development 

The  first,  and  most  critical,  opportunity  to 
successfully  manage  reliability  is  at  the  conception  of  a 
need.  This  is  the  beginning  of  the  system's  life  cycle 
and,  an  ideal  place  to  have  the  greatest  effect  by 
considering  reliability  in  the  original  design. 
Reliability  requirements  must  be  grounded  in  reality, 
accounting  for  system  usage,  expected  operating 
environment,  and  available  technology  and  resources.  The 
pursuit  or  acceptance  of  inadequate,  overstated,  or  simply 
inaccurate  reliability  targets  can  be  costly. 

The  1994  spec  and  standard  reform  not  only  made  a 
significant  change  in  document  usage,  it  directed  all 
acquisition  decision  makers  to  focus  on  the  "problem  rooted 
in  the  requirements  determination  phase."  [Ref.  34]  A 
fundamental  comprehension  of  the  ultimate  user' s  needs  is 
required.  A  system  to  be  operated  in  space,  aviation,  or 
in  a  security  function  demands  different  reliability 
requirements  than  does  an  administrative  computer  desktop 
operating  system  or  a  coping  machine.  Although  the 
ultimate  customer  wants  a  reliable  system  in  all 
circumstances,  the  consequence  of  system  failures  varies. 
Systems  to  be  operated  in  space  demand  high,  or  ultra, 
reliability.  The  cost  of  a  failure,  or  even  the 

opportunity  for  corrective  action,  is  prohibitive. 

Once  mission  and  environmental  profiles  have  been 
established,  the  intended  life  of  the  system  should  be 
determined.  This  has  been  a  problematic  issue  for  the 
Department  of  Defense,  but  needs  to  be  addressed,  as  this 
will  affect  a  new  system  from  the  materials  required  to  the 
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design  itself.  The  design  stage  also  presents  the 

opportunity  to  consider  designing  for  manufacturing  and 
producibility .  Engineers  tend  to  place  undue  emphasis 
specific  product  functions,  but  SEP  should  provide  a 
balance  that  can  help  maintain  perspective  during  design. 

Designing  reliability  into  a  system  can  be  achieved 
through  many  paths.  Techniques  such  as  maintaining 

simplicity  of  design,  or  the  selection  of  particular  raw 
materials  all  must  be  balanced  with  their  possible  side 
effects.  For  example,  if  a  design  team  chooses  to  optimize 
a  system' s  reliability  through  redundancy  of  critical 
components,  they  must  also  consider  the  affect  on  the 
logistical  burden  of  increased  part  storage  and 
maintenance.  There  are  a  multitude  of  tools  to  assist  in 
the  analysis  of  reliability  during  design. 

Below  is  a  non-inclusive  list  of  reliability  analysis 
tools.  Not  every  tool  or  technique  applies  to  every 
program.  Each  program,  it  is  recommended,  should  tailor 
their  selections  to  achieve  their  desired  results. 

•  Failure  Modes,  Effects,  and  Criticality  Analysis 
(FMECA) .  FMECA  is  a  method  to  identify  potential 
failure  modes,  causes  and  effects,  and  to  rank 
them  by  the  severity  of  consequences.  This  tool 
can  be  used  to  eliminate  or  minimize  potential 
problem  areas  or  at  a  minimum  provide  maintenance 
tasks  to  address  them. 

•  Parts  and  Material  Control  Program.  Parts 

control  enhances  standardization  and  reliability. 
Controlling  the  quality  of  materials/parts  used 
aides  in  accuracy  of  reliability  predictions 

•  Reliability  Prediction.  This  technique  uses 
parametric  estimations,  historical  or  engineering 
models  to  forecast  part  reliabilities. 
Prediction  estimates  are  a  useful  tool  but  should 
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only  be  used  when  it  is  followed  with 
verification  testing.  Although  predicted 
estimates  are  never  identical  to  fielded  results, 
there  are  numerous  examples  of  predicted  MTBFs 
that  were  not  even  close  to  the  actual 
performance . 

For  example,  the  F/A-18  contains  a  trailing  edge 
flap  actuator  that  controls  a  flight  control 
surface.  The  original  equipment  manufacturer 
rated  the  expected  MTBF  at  4000  flight  hours. 
The  actual  performance  of  this  equipment  turned 
out  to  be  less  than  400  hours.  Whether  the  cause 
of  the  error  was  from  incorrect  installation, 
unintended  usage  or  simply  from  inaccurate 
predictions  the  results  were  the  same.  The 
aircraft  availability  suffered  because  of  the 
unplanned  maintenance  required  and  the  supply 
system,  which  was  based  on  the  expected  failure 
rate,  was  unprepared  for  the  increased 
overwhelming  demand. 


A  hypothesis 

must  be  assumed  when 

dealing 

with 

predictive 

reliability : 

the 

estimate 

is 

incorrect  until  proven 

otherwise . 

The 

mere 

acceptance,  without  proof,  of  a  manufacturer's 
claim  is  unwise  at  best. 


•  Physics  of  Failure  (POF)  .  POF  is  a  method  that 

uses  modeling  and  simulation  to  identify 

mechanisms  of  failure  prior  to  testing.  This 
technique  can  help  focus  future  testing  and 
maintenance . 

•  Highly  Accelerated  Life  Testing  (HALT) .  HALT  is 

performed  on  a  product  as  part  of  the  design 
process.  A  product  is  stressed  beyond  its 

required  specifications  and  beyond  what  is 
expected  in  its  intended  environment.  This  is 


25 


done  to  induce  failures  and  identify  the  actual 
functional  and  destruct  limits  of  a  product. 
This  differs  from  tradition  verification  testing 
which  demonstrates  a  products  compliance  with 
specs.  HALT  exposes  the  weakest  points  in  the 
design  in  an  extremely  quick  fashion  with  a 
minimum  of  required  resources.  HALT  allows  early 
detection  and  correction  of  failure  points  that 
previously  would  not  have  been  discovered  prior 
to  fielding.  A  root  cause  failure  analysis  is 
completed  on  each  failure  allowing  for 
implementation  of  corrective  actions.  [Ref. 16] 

•  Stress  Analysis.  A  design  technique  enhanced 
with  computer-aided  engineering,  intended  to 
identify  effects  of  stress  on  a  system.  Through 
the  computer  the  stress  can  be  conducted  in  a 
worst-case  analysis.  This  analysis  can  help  to 
reduce  the  region  of  possible  failures  and 
decrease  the  nominal  stress. 


The  importance  of  the  design  phase  is  demonstrated  in 
Figure  5.  Nearly  eighty  percent  of  a  system's  life  cycle 
costs  are  committed  in  the  initial  stages  of  the  design 
phase.  Decisions  made  on  materials  and  processes  to  be 
used,  will  drive  the  rest  of  the  program.  Mistakes  or 
miscalculations  made  during  the  initial  design  stages  will 
put  the  program  office  in  a  reactive  position  for  years. 

Early  orientation  towards  reliability  may  put  a 
program  in  a  proactive  position.  Incorporation  of  built- 
in-test  functions  and  simple  designs  with  reliable  parts, 
will  contribute  to  the  ease  of  future  maintenance 
requirements.  The  inclusion  of  experts  from  multiple 
disciplines  early  in  process  will  identify  concerns,  which 
would  have  been  overlooked. 
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Figure  5.  Life  Cycle  Costs  [From  Ref.  37] 


2.  Tests,  Production  and  Verification 

A  reliability  program  continues  its  iterations  with 
each  stage  of  development.  The  outputs  from  the  previous 
iteration  become  the  inputs  to  the  present.  As  systems 
evolve  reliability  management  must  keep  pace.  Testing  and 
production  need  to  continue  emphasizing  reliability  and 
stress  reduction  on  the  system. 

The  reliability  targets  established  in  the  design  and 
development  stages  must  now  be  tested  and  verified.  The 
verification  of  accepted  targets  is  crucial.  The 
logistical  support  system  will  be  built  upon  the  accepted 
R&M  targets.  Production  process  must  also  be  examined.  It 
is  not  uncommon  for  the  production  to  introduce  faults  into 
the  product.  Testing  and  process  controls  are  the  key 
elements  to  success  at  this  point. 

The  production  process  seems  to  be  often  taken  for 
granted.  It  is  incumbent  upon  the  program  office  to  ensure 
that  methods  and  material  used  to  produce  a  product  do 
induce  reliability  problems.  Testing  and  inspection  of  the 
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manufacturing  process  must  be  completed  to  reduce 

opportunities  for  latent  defects  or  human  error. 

Test  and  evaluation  (T&E)  are  invaluable  to  a 
successful  program.  T&E  is  a  part  of  the  acquisition 
process  aimed  at  getting  the  best  possible  product  to  the 
warfighter.  Some  feel  that  testing  is  an  unnecessary 
burden  that  eats  into  both  cost  and  schedule.  T&E  does 
consume  cost;  both  in  terms  of  schedule  time  and  in 
financial  commitment.  However,  these  expenditures  serve 
two  primary  functions:  First,  to  help  develop  and  make  a 

system  work;  Second,  to  determine  if  a  system  works  and 
that  we  are  getting  what  we  asked  for.  There  are 
essentially  two  forms  of  testing  that  accomplish  these 
goals:  Developmental  Test  and  Evaluation  (DT&E)  and 

Operational  Test  and  Evaluation  (OT&E) .  Both  forms  of 
testing  provide  feedback  to  acquisition  decision  makers, 
a.  Developmental  Test  and  Evaluation 
DT&E  is  a  method  for  a  PM  to  make  his  system 
work,  to  verify  contractor  claims  and  predictions,  and  to 
influence  the  system  design.  DT&E  normally  aides  in 
product  design  and  development,  through  a  test-analyze-f ix- 
test  (TAFT)  approach.  Both  contractors  and  Governmental 
personnel  can  be  involved  in  each  of  these  four  stages. 
[Ref.  22] 

The  feedback  provided  by  DT  allows  the  systems 
engineering  process  to  analyze  the  test  results  and  devise 
required  fixes  or  adjustments  to  be  implemented  and  tested 
again.  Reliability  engineers  and  logistician  play  a  key 
role  through  the  IPT  process.  Ultimately  design  risks  are 
minimized  and  the  system  can  be  certified  ready  for 
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operational  testing.  Listed  below  are  some  of  the  testing 
techniques  and  tools  used. 

•  Failure  Reporting,  Analysis  and  Corrective  Action 

System  (FRACAS) .  FRACAS  is  a  closed-loop 

reporting  system  that  plays  an  integral  role  in 
TAFT.  The  formal  reporting  of  identified  issues 
allows  for  root  cause  determinations  and 

corrective  actions  to  be  applied.  Early 

initiation  of  FRACAS  provides  ample  time  to 
address  failures  prior  to  full-rate  production. 

•  Environmental  Stress  Screening  (ESS)  .  ESS  is  an 

initiative  to  find  potential  flaws  through  the 
use  of  thermal  cycling  and  vibrations  early  in 

the  production  process  enabling  cost  savings  by 
identifying  problems  in  the  factory  while  there 

is  time  to  fix. 

•  Highly  Accelerated  Life  Testing  (HASS) .  Hass  is 

a  screen  test  performed  during  the  production 
process.  This  test  is  to  be  completed  post  HALT 
and  DVT.  The  goal  of  HASS  is  to  ensure  that  no 
new  weak  link  has  entered  the  picture  during  the 
production  process.  [Ref.  16] 

•  Reliability  Development  Testing  (RDT) .  RDT  or 

reliability  growth  testing  (RGT)  is  the 

centerpiece  of  TAET .  The  reliability  growth  of 
the  system  is  emphasized  through  an  iterative 
design  maturation  process. 

The  PM  controls  the  DT  environment  and  is  provided 
with  data,  throughout  this  testing  cycle,  with  which  he  can 
make  informed  managerial  decisions  that  can  affect  the 
reliability  of  the  final  product.  Through  developmental 
testers  the  PM  must  review  contractor  reliability  programs, 
monitor  reliability  tests  and  review  test  data  and  reports. 
With  limited  resources  the  PM  must  make  tough  trade-off 
decisions  and  ultimately  certify  that  his  system  is  ready 
for  operational  test  and  evaluation  (OPEVAL) . 
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b.  Operational  Test  and  Evaluation 

OT&E  is  considered  the  conscience  of  the 


acquisition  process.  Unlike  DT,  operational  testing  is 
accomplished  by  an  independent  agency  not  beholden  to  the 
Program  Manager.  Testers  are  fleet  representative  users 
and  the  testing  environment  must  be  realistic  of  the 

intended  operational  environment,  including  anticipated 

threat  countermeasures.  OT&E  is  designed  to  stress  the 
system  as  it  will  be  used  by  the  warfighter. 

•  Early  Operational  Assessments  (EOA) .  EOA  is 
conducted  prior  to  Milestone  II/B  to  provide 
operational  /  mission  input  to  the  decision 
makers  early  in  the  life  cycle. 

•  Operational  Assessments  (OA) .  Operational 

assessments  begin  post  Milestone  II/B  and  signify 
the  start  of  the  OTAs  evaluation  of  system  level 
performance . 

•  OPEVAL .  OPEVAL  is  a  "separate  and  dedicated 

phase"  of  OT&E  conducted  in  support  of  the  full- 
rate  production  without  contractor  involvement. 
The  purpose  of  OPEVAL  is  to  evaluate  a  system' s 
Operational  Effectiveness  and  Suitability. 
Reliability  requirements  are  encompassed  within 
the  suitability  judgment,  and  the  OTA  will 
conduct  enough  testing  on  production 

representative  test  articles  to  make  that 
determination.  All  supporting  publications, 

logistical  support  planning  and  training  will 
also  be  evaluated. 


•  Follow-On  Test  and  Evaluation  (FOT&E) .  FOT&E  is 
testing  completed  after  milestone  III/C.  This 
testing  is  generally  used  to  test  modifications 
to  production  systems  or  to  complete  any  deferred 
testing . 


The  purpose  of  OT&E  can  be  simplified  to  the  main 
emphasis:  to  see  if  it  works.  In  essence  it  is  a  final 
exam  for  PMs,  and  the  last  chance  to  influence  and  evaluate 
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reliability  before  a  system  hits  the  fleet.  All 
reliability  targets  must  have  been  proven  within  a 
confidence  level  acceptable  to  the  decision  authorities. 

3.  Operations  and  Support 

The  program  office  is  responsible  to  sustain  the 
reliability  of  their  system  while  it  remains  in  service.  It 
is  in  this  stage  of  the  life  cycle  where  the  previously 
efforts  to  optimize  reliability  pay  off.  PMs  are  required 
to  "maintain  a  relationship  with  the  user/warf ighter  based 
on  system  readiness."  [Ref.  14] 

The  collection  of  field  data  is  essential  to  proper 
management.  The  only  true  measure  of  system  performance  is 
borne  of  that  systems  sustained  use  in  actual  conditions. 
Even  with  rigorous  testing,  experience  has  shown  that 
predicted  reliability  estimates  do  not  match  actual  system 
performance.  DoD  5000. 2-R  charges  the  PM  and  the  logistic 
community  to  measure  and  support  fielded  systems,  with 
emphasis  on  continued  improvement. 

Corrective  actions  must  be  taken  to  address  the 
differences  between  predicted  and  actual  performance.  If  a 
program  elected  to  spare  to  the  predicted  reliability 
levels  there  may  be  significant  delays  and  shortcomings 
within  the  available  maintenance  capabilities.  Adjustments 
must  be  made  either  within  the  system  itself  or  within  the 
supply  chain. 

F .  CHAPTER  SUMMARY 

The  reliability  of  a  weapon  system  directly  impacts 
upon  that  systems  operational  capability  and  life  cycle 
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costs,  making  it  of  fundamental  importance  to  the 
warfighter.  It  is  mandatory  for  all  program  managers 
within  the  Department  of  Defense  to  plan  for  and  execute 
measures  to  ensure  their  program  accounts  for  the  user' s 
RAM  objectives.  [Ref.  14] 

This  chapter  provided  a  framework  of  reliability 
definitions,  references,  and  methods.  Additionally,  the 
researcher  provided  a  view  of  both  reliabilities'  relative 
importance  in  system  acquisition  and  of  opportunities 
within  a  systems  life  cycle  for  a  PM  to  influence  the 
reliability  of  the  system  for  which  he  or  she  is 
responsible . 

Chapter  III  will  present  information  about  the  Naval 
Air  Systems  Command  programs  surveyed  for  this  thesis.  The 
results  of  the  survey  and  interviews  will  be  provided  and 
organized  by  applicable  themes. 
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III.  PROGRAM  METHODOLOGY  FOR  RELIABILITY  MANAGEMENT 


A.  INTRODUCTION 

This  chapter  presents  the  methodology  used  and  data 
gathered  to  address  the  primary  and  subsidiary  research 
questions.  An  overview  of  the  Naval  Air  Systems  Command 
(NAVAIR)  and  information  on  the  system  programs 
investigated  are  provided.  Additionally,  data  from  these 
programs  is  presented  in  an  aggregate  format  and  summarized 
based  on  the  responses  to  the  electronic  weapon  system 
reliability  management  survey  questionnaire. 

Data  presented  reflects  the  actions  and  perceptions  of 
the  acquisition  workforce  dealing  with  reliability 
management  issues. 

B .  METHODOLOGY 

Research  was  conducted  through  a  literature  search  of 
reliability  related  documents  including  Director  of 
Operational  Test  and  Evaluation  (DOT&E)  Reports,  General 
Accounting  Office  (GAO)  Reports,  Department  of  Defense 
(DoD)  Official  Memorandums  and  Directives,  Congressional 
Subcommittee  Reports,  Military  Standards  and  Handbooks  (Mil 
Specs),  and  technical  and  professional  journals,  manuals 
and  web  sites.  Interviews  were  conducted  with  current 
acquisition  professionals  familiar  with  program  and/or 
reliability  management  including  personnel  from  Program 
Executive  Offices  (PEO) ,  aviation  program  management  (PMA) 
offices,  contractors,  the  test  community,  user 
representative  organizations,  and  reliability  engineering 
disciplines.  Additional  interviews  were  conducted  with 
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personnel  from  academic  disciplines,  who  have  had  years  of 
experience  in  program  and  reliability  management. 
Interviews  were  conducted  in  person,  over  the  phone,  or 
through  electronic  mail. 

All  interviews,  literature  searches  and  the  survey 
questionnaire  were  aimed  at  the  primary  research  question: 

What  strategies  should  program  managers  implement  to 
optimize  inherent  reliability  in  their  weapon  system? 

1.  Survey  Questionnaire 

The  Weapon  System  Reliability  Survey  Questionnaire 
used  was  a  modified  version  of  a  previously  designed  survey 
intended  to  "draw  out  the  practices  employed  by  each  PM 
organization  on  managing  reliability  performance  risks  in 
their  programs."  [Ref.  39]  Electronic  distribution  was 
accomplished  with  the  assistance  of  the  Reliability  and 
Maintainability  Competency  at  NAVAIR.  Respondents  to  the 
questionnaire  included  reliability  engineers,  logisticians, 
operational  and  developmental  testers,  and  program 
management  personnel,  all  of  whom  had  responsibilities 
associated  with  reliability  within  their  specific  program. 
A  copy  of  the  survey  questionnaire  is  located  in  Appendix 
A. 

2 .  Interviews 

Interviews  were  conducted  as  an  additional  method  of 
addressing  the  research  question.  Follow-up  interviews 
with  survey  questionnaire  respondents  were  conducted  when 
amplification  to  their  inputs  was  desired.  Additional 
interviews  were  conducted  to  gain  insight  from  individuals 
who  had  experience  and  background  in  reliability  and 
management  issues.  The  empirical  data  from  interviewees 
was  generally  from  a  supervisory  perspective.  Generally, 
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these  people  consisted  of  PEO  personnel,  PMs,  competency 
branch  and  division  heads.  Interview  formats  focused 
primarily  on  the  thesis  research  questions,  which  the 
interview  subjects  normally  had  time  to  review  and  respond 
in  writing  if  desired. 

C .  PROGRAMS  STUDIED 

The  systems  researched  were  limited  to  Naval  Aviation 
programs  associated  with  NAVAIR.  NAVAIR  is  a  major  Naval 
Systems  Command  charged  with  systems  acquisition  and 
supporting  those  systems  in  the  operating  fleet.  Actual 
program  management  is  a  team  effort.  Program  Managers 
(PMs)  work  for  a  supervisory  Program  Executive  Office  (PEO) 
who  in  turn  reports  to  the  Assistant  Secretary  of  the  Navy 
for  Research,  Development  and  Acquisition  (ASN  (RDA) ) . 

The  ASN  (RDA)  is  responsible  and  accountable  for  all 
acquisition  functions  and  programs  for  the  Department  of 
the  Navy.  The  PEOs  and  PMs  are  directly  responsible  for 
the  development  and  acquisition  of  Naval  systems.  As  a 
Systems  Command,  NAVAIR  provides  matrix  support  to  PMs 
including  engineering,  contracting,  and  comptroller. 
Eigure  7  depicts  the  extensive  reporting  chain  for  NAVAIR. 

Thirteen  programs  spread  throughout  the  four  aviation 
PEOs  and  eleven  Aviation  Program  Management  (PMA)  offices 
participated  in  the  survey.  The  represented  systems  range 
from  pre-program  establishment  to  legacy  systems  deployed 
and  operated  for  over  20  years.  The  programs  with 
designated  Acquisition  Categories  (ACAT)  are  level  I  and 
level  II.  Perspectives  covering  additional  programs  were 
incorporated  through  the  survey  questionnaires  and 
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interviews  conducted  with  personnel  in 
positions  who  had  responsibility  and  knowledge 
programs . 


supervisory 
of  multiple 


Figure  6.  NAVAIR  Reporting  Chain  [From  Ref.  30] 


D .  DATA  PRESENTATION 

In  an  effort  to  obtain  disclosure  of  all  issues 
associated  with  reliability  management,  interviewees  and 
survey  questionnaire  respondents  were  permitted  to  provide 
information  under  the  premise  of  non-attribution.  Though 
most  participants  did  not  seem  concerned  with  attribution, 
the  "political  realities  of  system  acquisition"  led  some 
individuals  to  request  that  they,  or  their  program,  not  be 
identified.  As  such,  all  program  responses  will  be  treated 
in  the  aggregate.  Participating  programs  and  individual 
research  participants  will  not  be  identified,  as  it  would 
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be  too  difficult  to  maintain  anonymity  for  some  while 
identifying  others. 

Collected  data  to  be  analyzed  will  be  organized  in  the 
following  themed  categories: 

•  Reliability  Management  Environment. 

•  Reliability  Processes  and  Tools. 

•  Reliability  of  Fielded  Systems. 

•  Affects  of  Acquisition  Reform. 

The  following  sections  will  present  the  aggregate  data 
within  the  same  areas  of  interest.  The  data  summary 
includes  information  from  all  sources  of  data  collection. 
Where  appropriate  tables  will  be  used  to  display 
questionnaire  responses.  [Ref.  39] 

E.  RELIABILITY  MANAGEMENT  ENVIRONMENT 

The  first  set  of  data  relates  to  how  management 
approaches  the  subject  of  reliability  and  seeks  to  identify 
the  cultural  environment  of  reliability  management.  A 
series  of  survey  questionnaire  questions  were  intended  to 
draw  out  the  perceptions  of  respondents  on  how  reliability 
is  managed. 

1 .  Reliability  Responsibility 

The  results  tabulated  in  Table  2  present  the 
questionnaire  answers  to  the  following  question: 

Who  within  your  organization  is  primarily  responsible 
for  reliability  activities  for  this  particular  program? 

This  question  was  fundamental  in  establishing  whether 
there  was  a  consistent  managerial  approach.  The  responses 
are  presented  below: 
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Responsible  for  Reliability  Within 
PMA 

Total 

0, 

O 

Chartered 

IPT? 

Y 

N 

PM 

1* 

Project  Leader 

Systems  Engineering  Team  Lead 

1  (1*) 

8% 

Logistics/Supportability  Team  Lead 

4 

31% 

Test  Team  Lead 

Reliability  IPT 

5 

38% 

4 

1 

Prime  Contractor 

1 

8% 

Lead  Shared 

2 

15% 

No  One  Specifically 

*  indicates  conflicting  response 

Table  2.  PMA  Reliability  Responsibility 


Response  Summary:  Responses  varied  throughout  the 
programs  without  any  unified  theme.  The  largest  common 
response  indicated  that  the  Reliability  IPT  had  been 
delegated  primary  responsibility  for  reliability  issues. 
However,  less  than  half  of  the  participating  programs  even 
had  a  formal  Reliability  IPT.  Additionally,  two  program 
responses  could  not  identify  an  individual  or  team  that  had 
overarching  authority  over  reliability  activities.  Instead 
they  indicated  that  reliability  management  did  not  have  a 
designated  primary  authority ,  and  was  purposely  a  shared 
responsibility  through  multiple  sources. 

One  program  provided  multiple  responses  to  the 
questionnaire  inclusive  of  the  following  perspectives: 
reliability  engineer,  subsystem  IPT  lead,  and  tester.  Each 
individual  response  identified  a  different  source  of 
"primary  responsibility."  The  PM,  Logistics/Supportability 
Team  and  Reliability  IPT  were  each  identified  as  having 
primary  responsibility  for  reliability. 
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Additional  Comments:  During  an  interview,  one  former 
PM  commented  that  PMs  manage  by  exception  and  without  a 
specific  problem  or  issue,  reliability  and  the  other 
engineering  disciplines  are  managed  through  empowerment  of 
the  technical  experts.  The  common  theme  seemed  to  be  an 
acknowledgement  that  PMs  rely  upon  the  reliability 
competency  for  matrixed  support.  Reliability  experts 
generally  provide  input  on  what  reliability  activities  are 
suggested  and  where  they  should  be  implemented.  How  that 
support  is  incorporated  into  a  program  is  dependent  on 
available  funding  and  the  PMs  judgment  based  on  cost, 
schedule,  technical  risk  and  political  environment 
considerations . 

2 .  Reliability  Dociamentation 

Most  acquisition  activities  have  an  overriding  program 
document  that  provides  structure,  priorities,  methodology 
and/or  resources  for  a  given  topic.  For  example,  the  Test 
and  Evaluation  Master  Plan  (TEMP)  is  the  overarching 
document  relating  all  test  activities  within  a  program. 

How  is  the  system  reliability  program  and 
corresponding  management  approach  to  such  formally 
dociamented? 


Table  3  provides  the  survey  questionnaire  responses. 


Reliability  Documentation  Within  PMO 

Program 

Responses 

%  of 

Programs 

Reliability  Program  Plan 

9  69% 

Contract  Statement  of  Work  (SOW) 

3  23% 

Test  and  Evaluation  Master  Plan  (TEMP) 

5  38% 

Single  Acquisition  Management  Plan  (SAMP) 

0 

No  Formal  Reliability  Management  Plan 

1  8% 

Other 

1  8% 

Table  3.  Formal  Reliability  Documentation 
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Response  Summary:  The  majority  of  participating  Naval 


Aviation  programs  do  indeed  have  an  overarching  reliability 
document.  Only  one  program,  still  in  the  earliest  stages 
of  development,  did  not  yet  have  any  formal  reliability 
management  plan.  In  addition  to  a  Reliability  Program 
Plan,  many  programs  track  reliability  activities  through 
the  TEMP  or  Statement  of  Work  (SOW) . 


3.  Reliability  Resources 


Allocation 

of  available 

resources 

is  inherently 

a 

management  function. 

As 

such,  participating 

PMA 

representatives 

were 

asked 

to  assess 

the  adequacy 

of 

reliability  resources. 

Table 

4  provides 

the  responses 

to 

the  following  question: 

Is  the  amount  of  time  and  funding  allotted  for 
reliability  testing  sufficient  for  your  program? 


ADEQUACY  OF  RELIABILITY  RESOURCES 

Current  Schedule  and 
Available  Funds  are 
Sufficient 

Could  Use  More 
Time/$$  to 
Reduce 

Reliability  Risk 

No  Significant 
Reliability 
Effort  at  This 
Time 

4 

6 

3 

31% 

46% 

23% 

Table  4.  Adequacy  of  Reliability  Resources 

Response  Summary:  The  PMAs  that  responded  that  their 
program  had  sufficient  funding  fit  into  two  categories;  1) 
High  visibility  programs  early  in  the  development  stage 
enjoying  positive  support  and  funding  or,  2)  Legacy 
programs  enjoying  considerable  support  in  part  due  to  their 
current  tactical  usage. 
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The  PMAs  that  responded  that  they  currently  were  not 
engaged  in  significant  reliability  efforts  were  either 
early  in  the  defining  requirement  stage  or  have  just 
successfully  entered  fielding  after  making  required 
adjustments  to  production  articles.  The  remaining  programs 
fit  the  more  common  profile  of  a  program  competing  for 
funding  and  undergoing  the  scrutiny  that  is  a  part  of  the 
acquisition  process.  Knowing  that  resources  are  limited, 
tough  decisions  often  lead  to  compromise  on  schedules  and 
priorities  effecting  system  reliability. 

Additional  Comments :  One  Reliability  IPT  member 
stated  that  reliability,  maintainability  and  supportability 
are  always  the  elements  that  are  compromised  when  executing 
a  program.  He  further  commented  that  these  activities  are 
so  essential  that  they  need  to  be  fully  funded  and  not  have 
their  schedules  so  compressed  that  testing  is  either 
severely  cut  or  meaningless. 

An  interviewed  PM  said  that  program  management  often 
operates  in  the  gray  area,  and  that  we  knowingly  "mortgage 
the  O&M  future"  to  gain  production  support  today.  A 
reliability  IPT  team  leader  provided  the  following 
discussion  on  reliability  funding: 

Reliability  costs  are  largely  up  front  in  the 
design  and  testing  phases  and  can  be  rather 
significant....  When  we  come  to  testing  and 
fielding  and  we  find  shortfalls  in  the  system 
reliability.  Normally  the  cost  to  make  any 
reliabilty  improvemnts  comes  from  the  same  pot 
[of  money]  as  system  performance  improvements, 
which  usually  win  out  over  reliability.  The 
programs  need  to  have  a  contingency  fund  for 
reliability  and  maintainability  fixes.  Early  in 
the  program  when  there  are  few  units  in  the  field 
this  would  have  a  very  good  chance  of  providing  a 
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quick  payback  in  logisics  savings. 


The  general  consensus  of  comments  can  be  summed  up 
with  the  following  quote: 

We  will  either  pay  for  it  up-front  or  the  fleet 
will  pay  for  it  in  terms  of  reliability  or 
maintainability  once  it  gets  fielded  in  the 
fleet . 

4 .  Reliability  Regulations  and  Policies 

Regulatory  documentation  is  a  reflection  of  the 
attention  paid  to  a  subject  by  higher  headquarters. 

Are  you  aware  of  any  specific  DoD  or  Navy  policy 
regulation  regarding  weapon  system  reliability  management? 

Table  5  summarizes  the  responses. 


Reliability 

Policy 

Awareness? 

Program 

Responses 

0, 

o 

YES 

4  31% 

NO 

4  31% 

NOT  SURE 

5  38% 

Table  5.  Reliability  Policies  and  Regulations 
Response  Summary: 

Programs  that  replied  yes  identified  the  DoD  5000. 2-R, 
specific  program  documentation,  or  the  cancelled  military 
specifications  and  standards.  Nine  out  of  13  programs,  and 
12  out  of  18  questionnaire  responses,  stated  that  they  were 
unaware  or  unsure  of  any  policy  or  regulation  regarding 
reliability  management.  Most  responses  and  interviews 
further  commented  on  frustration  with  a  lack  of  useful 
documented  guidance. 
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Additional  Paraphrased  Comments: 


The  following 


comments  are  paraphrased  responses: 

•  DoD  5000.2  and  not  much  more;  with  acquisition 
reform  the  Government  backed  off  military  specs 
and  standards. 

•  There  has  been  a  reduction  in  the  design 
fundamentals  [MIL-STD/HNDBK]  throughout  the  90' s. 

•  Some  PMAs  were  told  to  shift  to  commercial  best 
practices,  unfortunately  the  contractor  does  not 
always  know  better. 

•  Reliability  engineers  continue  to  use  the 
cancelled  military  specs  and  handbooks  because 
there  has  been  nothing  to  replace  them. 

•  Some  reliability  engineers  just  cut  portions  of 
MIL-STD-785/781  and  paste  them  into  the  Statement 
of  Work  (SOW) . 

•  At  program  initiation  the  Willoughby  reliability 
improvement  initiatives  were  included  in  the 
specification . 

•  "If  you  identify  some,  please  let  me  know." 

F.  RELIABILITY  TOOLS,  TECHNIQUES  AND  PROCESSES 

This  section  provides  information  on  the  methods  and 
strategies  used  to  implement  reliability  management.  Data 
pertaining  to  the  use  of  contractual  incentives  will  be 
discussed  along  with  the  issues  of  influencing 
requirements,  designing-in  reliability,  and  test  and 
evaluation . 

1 .  Requirements  Generation 

Requirement  generation  is  the  genesis  of  every 
acquisition  program.  The  data  presented  here  was  gathered 
to  address  the  PM' s  ability  to  influence  system 
requirements,  with  respect  to  reliability,  during  their 
initial  formation.  Programs  were  asked: 
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Were  you  as  the  material  developer  able  to  influence 
the  incorporation  of  realistic  reliability  requirements  as 
part  of  the  ORD  process? 

Table  6  depicts  the  responses. 


Ability  to 
Influence 
Reliability 
Requirements  in 
the  ORD? 

Program 

Responses 

o 

o 

YES 

9 

69% 

NO 

2 

15% 

Other 

1 

8% 

Table  6.  PMs  Influence  on  Requirement  Generation 

Additionally  PMAs  were  asked  whether  reliability  was 
identified  as  a  key  performance  parameter  (KPP) .  Because 
KPPs  are  those  requirements  or  capabilities  that  are  deemed 
to  be  so  critical  that  failure  to  reach  the  threshold  level 
may  result  in  the  programs  termination,  the  researcher 
wanted  to  determine  if  reliability  issues  reached  this 
level  of  scrutiny.  Only  four  programs  answered  in  the 
affirmative,  with  others  replying  that  although  not  a  KPP, 
reliability  ranks  within  the  highest  tier  of  priority. 

Response  Summary:  Overwhelmingly  the  programs 
participated  in  the  derivation  of  requirements  to  be 
included  in  the  Operational  Requirements  Document  (ORD) . 
The  "no"  votes  reflect  legacy  programs  in  which  a  sister 
service  acts  as  the  executive  agent  and  whose  original 
requirements  document  cannot  be  located.  The  "other"  vote 
is  from  a  support  equipment  program  that  gets  its 
requirements  through  the  systems  it  supports. 
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However,  there  was  not  unanimity  in  support  of  the 
requirement  process.  One  team  leader  felt  that  the 
original  user  requirements  were  altered  not  because  of 
technical  necessity  but  to  merely  to  satisfy  the  desired 
contractor's  estimates.  The  natural  course  of  compromise 
that  occurs  in  requirements  generation  is  hard  to  quantify 
in  terms  of  severity,  because  the  consequences  are  not 
realized  until  some  point  in  the  future. 

Supporting  Comments :  The  following  comments  are 

paraphrased  from  survey  questionnaire  responses  and 
interviews . 

•  Initial  values  were  solicited  from  operational 
users  and  requirement  'gate  keepers,  '  but  were 
modified  to  meet  potential  contractor  estimates 
based  on  financial  incentives  and  contract  award 
fees . 

•  User  requirements  are  sometimes  in  need  of  a 

reality  check.  The  reliability  engineering 

realities  do  not  support  the  uninformed  user. 

•  If  reliability  experts  are  not  integrated  into 
the  IPTs  until  after  the  ORD  and  contracts  are 
developed,  it  will  take  years  to  correct. 

•  R&M  is  not  a  KPP,  but  is  much  on  the  minds  of  the 
PMA  in  relation  to  passing  OPEVAL . 

2 .  Design  for  Reliability 

An  ideal  time  to  consider  reliability  of  a  system  is 
while  there  are  still  opportunities  to  influence  the 
outcome.  Given  a  set  of  agreed  upon  requirements,  programs 
must  then  translate  those  requirements  into  a  functioning 
system.  With  that  in  mind,  the  survey  questionnaire 
participants  were  asked: 

What  contractual  design  tools  were/are  employed  to 
ensure  reliability  is  "built  in"  early  on  in  the  program? 


45 


Table  7  depicts  the  program  responses. 


Types  of  Design  Tools 
Used  to  "Design-in" 
Reliability  Upfront  in  a 
Program 

Program 

Responses 

q, 

o 

Physics  of  Failure  (PoF) 

2 

15% 

Critical  Items 

List /Analysis 

6 

46% 

Identification  of  Known 
Problem  Areas 

9 

69% 

Software  Reliability 
Assessment 

7 

54% 

Quality  Function 
Deployment 

I 

8% 

Parts  Control  Program 

8 

62% 

FMECA  /FRACAS  /FTA 

12 

92% 

Reliability  Prediction 
Analysis 

4 

31% 

Table  7 . 


Reliability 


Design 


Tools 


Response  Summary:  All  DMAs  reported  using  design 

tools  to  address  reliability  within  their  system.  100 
percent  of  the  programs  use  some  form  of  failure  analysis 
as  an  integral  part  of  design.  There  was  a  clear  consensus 
that  employment  of  tools  that  incorporate  reliability 
considerations  into  system  design  was  required. 

Additional  Paraphrased  Comments:  The  following 

paraphrased  comments  reflect  the  survey  questionnaire 
responses  describing  additional  tools: 

•  Specification  Allocation  -the  [MTBF]  requirement 
was  allocated  to  each  element  of  the  weapon 
system. 

•  Design  to  Allocation  -  required  the  minimum 
design  to  allocation  to  be  at  least  25%  above  the 
specification  allocation  to  ensure  confidence  in 
achieving  specified  requirements. 

•  Design  and  Fleet  Field  predictions  were  used. 
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•  Stress  analysis  and  stress  derating  were  also 
done . 

•  Implemented  prediction  that  directly  correlates 
to  other  program  goals  and  have  driven  contractor 
to  that  prediction. 

•  As  reliability  engineers  we  list  all  of  tools 
that  we  think  will  be  useful,  knowing  that  PMs 
will  cut  many  of  them  citing  fiscal  restraints. 

3.  Contracting  for  Reliability 

Contracts  and  contractual  incentives  are  often  used  as 
motivation  tools  or  strategies  attempting  to  focus  a 
contractor's  effort.  Contracts  produce  a  similar  effect  on 
the  Government's  side  by  concentrating  attention  on 
particular  components  of  a  given  acquisition  program. 
Several  question  posed  in  the  survey  questionnaire  were 
aimed  at  determining  how  reliability  requirements  are 
interpreted  into  a  contract. 

The  first  contract  related  question: 

Was  reliability  included  as  a  factor  in  source 
selection? 

Table  8  depicts  the  survey  responses. 


RELIABILITY 

SOURCE 

AS  A  FACTOR  IN 

SELECTION 

YES 

NO 

UNK 

6 

4 

3 

46% 

31% 

23% 

Table  8.  Reliability  in  Source  Selection 

Response  Summary:  Half  of  the  programs  confirmed  the 
use  of  reliability  measures  during  source  selection.  These 
programs  generally  represented  ACAT  I  high  dollar  programs 
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with  large  defense  contractors.  In  general  most 
respondents  felt  that  issues  like  FMECA,  FRACAS,  and 
predictions  were  discussed  even  if  not  formal  source 
selection  criteria. 

The  second  contract  related  question  inquired  about 
the  translation  of  operational  requirements  to  contractual 
requirements.  Table  9  provides  the  answers  to  the 
following  question: 

How  are  ORD  reliability  requirements  for  your  program 
translated  into  actual  contractual  reliability 
requirements  ? 


TRANSLATION  OF  ORD  RELIABILITY 
REQUIREMENTS  TO  CONTRACTUAL 
REQUIREMENTS 

ORD  Requirement  Restated 
in  SOW 

6 

Additional  Levels 

Applied  to  Contract 

4 

Reliability  Not 

Adequately  Addressed 

1 

Other 

1 

Table  9.  Reliability  Requirements  in  the  Contract 

Response  Summary:  All  of  the  programs  that  have 
established  contracts  stated  that  at  a  minimum  the  ORD 
requirements  were  restated  in  the  SOW.  There  was  one 
dissenting  comment  that  pertained  to  degradation  of 
operational  requirements  being  nipped  at  in  small 
increments  beginning  with  their  translation  into  the 
contract . 
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Additional  Paraphrased  Comments: 

•  As  part  of  the  IPT  process  the  contractor  was 
tasked  with  developing  a  R&M  design  program  that 
would  reduce  risk  and  demonstrate  the  ORD 
requirements . 

•  The  R&M  audit  trail  process  dictates  the  process. 

ORD  values  are  translated  into  Design 

Controllable  numbers,  with  confidence  bands  and 
safety  margins  applied  to  go  from  ORD  to  TEMP  to 
SPEC. 

•  Design  controllable  reliability  requirement  added 
to  spec  /  ORD  had  value  that  included  induced 
failures . 

•  The  primary  metric  for  assessing  operating 
reliability  of  the  [system]  is  the  Captive  Carry 
Mean  Time  Between  Eailure  (MTBF) .  The  [system] 
reliability  is  heavily  dependant  on  the  host 
aircraft  upon  which  it  is  being  employed  and  the 
corresponding  environment  in  which  the  aircraft 
is  flown. 

•  [System]  variants  have  been  deployed  for  a  number 
of  years  in  the  USN/USAE  inventory  and  captive 
carry  MTBE  has  been  determined  to  be  the  best 
reliability  metric  based  on  its  use  history. 
Therefore,  a  single  MTBE  number  was  detailed  in 
the  SOW/perf ormance  specification,  which  was 
provided  to  the  contractor. 

•  New  programs  are  putting  the  [reliability]  issues 
in  the  contract  as  a  means  of  enforcement. 


Lastly,  PMAs  were  asked: 

Are  there  incentives  employed  in  the  contract  that  are 
specifically  tied  to  achieving  system  reliability 
performance  requirements? 

Their  responses  are  summarized  in  table  10. 
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Are  Reliability 
Incentives 
Incorporated 
Within  the 

Contract? 

Program 

Responses 

If  Yes,  Did  The 
Incentives  Achieve 
Their  Desired  Effect? 

Y 

N 

Too  Early  to  Tell 

Yes 

5 

5 

No 

7 

Table  10.  Reliability  Contractual  Incentives 

Response  Summary:  With  less  than  half  of  the  programs 
Indicating  that  they  have  Incorporated  Incentives  In  the 
contract,  there  was  a  mixed  response  to  this  question. 
Some  programs  responded  that  reliability  goals  were 
achieved  through  IPT  style  supervision,  rather  than  actual 
contractual  Incentives.  None  of  the  programs  that  have 
established  reliability  Incentives  have  yet  had  an 
opportunity  to  evaluate  effectiveness. 

Additional  Comments:  One  technical  team  lead,  with  an 
operational  background.  Indicated  that  the  Incentives  on 
his  program  were  unfortunately  misguided.  He  felt  that  the 
Incentives  shifted  the  focus  from  operational  pertinent 
reliability  requirements  to  ones  based  on  contractor 
estimates  of  "achievable  standards'  not  reflective  of 
original  agreed  upon  requirements.  Additionally  he  stated: 

We're  100%  over  budget  and  not  meeting  the  early 
DT  testing  conditions.  Fundamental  shortfalls 
programmatically  In  design,  systems  engineering 
and  simulation  have  produced  an  Immature  design 
held  to  financial  constraints. 

From  the  same  program  another  respondent  reported  that 
during  the  design  phase  R&M  was  an  Award  Fee  consideration, 
but  was  outranked  by  cost  and  schedule  Impacts.  During 
testing  reliability  Is  being  measured  against  Its 
contractual  requirements. 
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4.  Reliability  Test  and  Evaluation  Activities 

Test  and  evaluation  (T&E)  is  an  integral  part  of  every 
program.  T&E  is  undertaken  both  to  aid  in  the  development 
of  a  system  and  to  verify  that  a  given  product  meets  the 
standards  as  stated  by  the  customer.  This  data  section  was 
derived  from  a  series  of  questions  that  addressed 
reliability  growth  programs,  common  understanding  of 
reliability  terms  among  parties,  test  activities,  and 
reliability  entrance  criteria. 

a.  Reliability  Growth  Program 

Most  programs  incorporate  a  Test-Analyze-Test-Fix 
(TAFT)  approach  to  product  development.  An  essential 
reliability  component  of  TAFT  is  the  reliability 
development  or  growth  plan  (RDP/RGP) .  Survey  questionnaire 
respondents  were  asked  whether  their  program  had 
incorporated  a  RGT .  Table  11  depicts  the  responses  to  the 
following  question: 

Does  your  program  incorporate  a  reliability 
growth  program? 


Does  Your 
Program 
Incorporate  a 
Reliability 
Growth  Program? 

Program 

Responses 

0, 

o 

Passed  Reliability 
Requirement  in  RQT  or 
Initial  OT? 

Y 

N 

Did  Not  Have 

Yet 

Yes 

7 

54 

2 

3 

No 

6 

46 

1 

4 

N/A 

1 

8 

1 

Table  11.  Incorporation  of  Reliability  Growth  Program 


Response  Summary:  Most  of  the  participating  programs 

have  an  established  reliability  growth  program.  The  "not 
applicable"  response  was  from  an  acquisition  program  to 
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early  in  its  life  cycle  to  have  yet  established  a  RGP . 
Overall  the  response  was  mixed  between  the  PMAs  with  formal 
growth  programs  and  the  PMAs  who  claim  to  achieve 
associated  reliability  growth  in  the  natural  course  of 
development . 

Additional  Paraphrased  Comments:  The  following 

comments  are  paraphrased  from  interview  and  survey 
respondents : 

•  Both  the  Statement  of  Work  and  the  contractor' s 

Reliability  Development  Growth  Program  Plan 

outline  our  reliability  growth  tests. 

•  In  the  design  specification  in  the  form  of  a 

reliability  requirement  to  be  demonstrated  in  a 
Technical  Evaluation  period  at  the  end  of  the  EMD 
testing  prior  to  OPEVAL . 

•  No  reliability  growth  programs  although  technical 

directives  are  issued  to  correct  known 

deficiencies . 

•  Weapon  systems  have  to  grow  into  it 

[reliability] . 

•  PMs  may  need  to  add  developmental  tests  to 

achieve  the  required  level  of  confidence. 

b.  Common  Terms  and  Methods  for  Measuring 
Reliability 

Common  understanding  of  terms  is  fundamental  to  a 
mutual  understanding  of  reliability  test  results.  This  is 
an  issue  when  comparing  different  weapon  systems  (e.g.  F/A- 
18  vs.  JSF) ,  a  lack  of  agreement  amongst  cooperative 
partners  within  a  program  can  be  paralyzing.  Additionally, 
there  is  a  difference  between  how  the  operator/user  views 
and  measures  reliability,  and  how  reliability  is  measured 
during  developmental  testing  within  the  same  program.  Each 
of  the  PMAs  was  asked: 
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Have  the  user,  tester,  contractor  and  PMO  all 
agreed  upon  the  method  (model)  to  be  used  in  reliability 
calculations  ? 

Table  12  depicts  the  results. 


Have  the  PM,  User, 
Contractor,  and  Tester 
Agreed  Upon  the  Method  to 
be  used  for  Measuring 
Reliability  ? 

Program 

Responses 

0, 

o 

Yes 

9 

69% 

No 

1 

8% 

Not  Sure 

3 

23% 

Table  12.  Common  Agreement  on  Reliability  Methods 

Response  Summary: 

Most  of  the  responding  PMAs  acknowledged  a  positive 
effort  to  reach  a  common  understanding.  The  sole  "no 
response"  was  from  the  program  that  has  not  yet  established 
their  methods  or  terms.  The  "not  sure"  responses 

correspond  to  respondents  who  stated  that  they  were  not 
familiar  with  this  particular  issue  because  the  test  phase 
occurred  years  before  they  became  involved  in  the  program. 

Additional  Paraphrased  Comments: 

•  ORD,  SPEC  and  TEMP  have  standardized  R&M  formulas 
and  definitions. 

•  The  contract  requirement  deals  with  design 

controllable  or  inherent  failures.  To  provide 
the  fleet  perspective  the  test  data  collected  and 
scored  using  specific  criteria,  the  results  could 
be  presented  as  to  how  the  system  is  performing 
in  terms  of  contract  requirement  and 

simultaneously  present  the  expected  fleet 
performance . 

•  The  Joint  Reliability  and  Maintainability 
Evaluation  Team  and  Test  Data  Scoring  Board 
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(JRMET/TDSB)  charter  documents  the  methods  to  be 
used . 

•  Captive  Carry  MTBF  is  specified  in  the  contract. 

•  The  NAVAIR  model  was  agreed  to  and  its  use  is 
stated  in  our  reliability  attachment. 

c.  Reliability  Test  Activities 

Reliability  testing  can  take  many  forms  and  the 
wide  selection  of  possible  techniques  allows  for  tailoring 
a  test  program  to  fit  a  given  program.  The  following 
question  was  aimed  to  determine  which  test  activities  PMAs 
use . 

Identify  the  types  of  test  activities  that  have 
or  will  be  used  to  determine  compliance  as  part  of  your 
system's  reliability  program. 


Types  of  Test  Activities  PMs  Use  to 
Determine  Reliability  Performance 
Progress  &  Compliance 

Program 

Responses 

0, 

o 

Environmental  Testing/Stress  Screening 

11 

85% 

Accelerated  Testing  (e.g.  HALT) 

7 

54% 

Reliability  Development  Growth  Test 
(RDGT) 

6 

46% 

Reliability  Qualification/ 
Demonstration  Test  (RQ/DT) 

5 

38% 

Government  Development  Test  (DT) 

7 

54% 

Operational  Testing  (e.g.  LUT/ 
OPTEMPO/IOTE/FOTE 

10 

77% 

Acceptance  Test/Production 

Verification  Test 

Maintenance  Demonstration 

Table  13.  Reliability  Test  Activities 

Response  Summary: 

Programs  overwhelmingly  address  reliability  through 
environmental  screening  and  operational  testing. 
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Additionally  some  programs  reported  the  use  of  software 
functional  testing  and  stockpile  reliability 
testing/ sampling . 

d.  Reliability  Entrance  Criteria  for  OPEVAL 

OPEVAL  is  the  final  exam  for  programs  just  prior 
to  the  decision  on  whether  the  system  will  go  on  to  full- 
rate  production.  Given  the  importance  of  OPEVAL,  and  the 
history  of  programs  having  significant  shortfalls  in 
reliability  during  this  phase  of  testing,  each  of  the  PMAs 
were  asked  about  their  approach  to  addressing  this 
situation.  [Ref. 2  DOT&E,  00]  Table  14  summarizes  the  PMA 
responses  to  the  following  question: 

Does  (or  did)  your  program  have  specific  OPEVAL 
entrance  criteria  relative  to  reliability? 


Does  Your  Program 

Have  Reliability 

Program 

g. 

Entrance  Criteria 

Responses 

for  OPEVAL? 

Yes 

4 

31% 

No 

5 

38% 

Not  Sure 

4 

31% 

Table 


Reliability  Entrance  Criteria 


Response  Summary: 


Surprisingly,  only  one-third  of  the  programs  surveyed 
have  established  dedicated  reliability  entrance  criteria  to 
OPEVAL.  Some  of  these  programs  have  even  identified 
entrance  criteria  for  the  earlier  phase  of  operational 
assessment  (OA)  .  Eor  example,  OA  entrance  criteria  were 
25%  of  the  OPEVAL  requirement  allowing  for  immaturity  of  a 
system  in  its  early  stage  of  development. 
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e.  Early  Test  Results  versus  OPEVAL 

Testing  within  a  program  occurs  on  a  continuum 
beginning  with  early  contractor  testing  and  progressing 
through  OPEVAL  into  follow-on  test  and  evaluation.  The 
purpose  of  development  testing  is  to  help  mature  a  system 
and  to  verify  that  the  product  does  indeed  meet  technical 
expectations.  OPEVAL  is  focused  on  the  operational 

effectiveness  and  operational  suitability,  and  is  conducted 
outside  of  the  restrictive  bounds  of  the  DT  environment. 
The  PMAs  were  asked  if  the  early  testing  related  to  success 
in  OPEVAL. 


If  your  system  has  already  participated  in 
OPEVAL,  did  your  success  in  either  DT  or  other  reliability¬ 
testing  correlate  with  success  in  OPEVAL? 

Table  15  relates  the  program  responses  to  OPEVAL, 


as  well  as  emerging  DT/OT  results. 


Level  of  ORD 
Reliability 
Requirement 
Demonstrated 

Correlation  of  Early 
Reliability  Test  Results  with 
OPEVAL  Results? 

Program 

Response 

0, 

o 

Initial 

DT 

Results 

Initial 

OT 

Results 

Yes,  success  in  pre-OPEVAL 
reliability  testing  led  to 
requirements  being  fully  met 
in  initial  lOTE. 

3 

23% 

100% 

100% 

4 

4 

Not  completely,  system  did 
well  in  early  testing  but  had 
some  problems  in  OPEVAL 

2 

15% 

80% 

80% 

1 

1 

Not  at  first,  system  passed 
OPEVAL  after  X  attempts. 

60% 

60% 

1 

N/A,  system  either  not  yet 
involved  in  an  operational 
test  or  the  early  testing  did 
not  assess  reliability. 

8 

62% 

<  40% 

<  40% 

1 

Table  15.  Correlation  of  Early  Test  Results  with  OPEVAL 
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Response  Summary: 


The  majority  of  the  responding  programs  had  not  yet 
undergone  the  scrutiny  of  OPEVAL .  Those  programs  that  had 
experienced  OPEVAL  were  evenly  split  on  their  responses. 
Reliability  success  in  pre-OPEVAL  testing  does  not  assure 
success  in  OPEVAL  itself.  Programmatic  success  relates  to 
how  the  PMA  used  the  information  gained  from  the  testing 
undertaken,  and  under  what  conditions  or  environment  the 
system  was  tested. 

Additional  Paraphrased  Comments :  Several  respondents 

included  illustrative  examples  from  their  programs.  Here 
are  some  paraphrased  examples: 


•  Some  had  a  combined  DT/OT  test  (OT-IIA)  ,  which 
used  EDM  configuration  hardware  with  encouraging 
results . 

•  In  general  systems  that  completed  qualification, 

reliability,  and  flight  tests  then  had  corrective 
actions  installed  and  retested  did  well  in 

OPEVAL.  In  a  number  of  cases,  systems  underwent 

process  changes  or  had  improvements  incorporated 
but  were  not  flight-tested  and  experienced  a  lot 
of  problems  in  OPEVAL  due  to  infant  mortality  and 
bad  process  control. 

•  TAET  was  painful,  but  it  seemed  to  work.  We 
instituted  a  corrective  action  board  that  tracked 
identified  problems  and  approved  solutions  and 
determined  the  degree  of  retesting  required. 

•  The  primary  lesson  learned  on  [system]  was  that  a 
tactically  operationally  representative  test 
program  where  the  test  article  can  be  tested  in 
the  full-up  configuration  on  the  aircraft  and 
functionally  tested  almost  real-time  is  the  most 
effective  method  for  assessing  inherent 
reliability  and  the  probability  of  passing  formal 
OPEVAL . 
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G.  RELIABILITY  OF  FIELDED  SYSTEMS 

Programs  that  have  properly  planned  and  executed 
effective  reliability  management  plans  may  reap  benefits 
once  a  system  reaches  the  hands  of  the  war  fighting 
operators.  History  has  shown  that  in  practice,  however, 
that  the  PMA' s  reliability  work  is  not  yet  complete.  The 
DoD  5000. 2-R  specifically  charges  PMs  and  the  logistics 
community  with  supporting  fielded  systems.  [Ref.  14] 

This  section  seeks  to  provide  information  on  how  this 
support  mission  is  accomplished.  Only  two  of  the  thirteen 
programs  participating  in  the  survey  questionnaire  had 
reached  the  fleet  fielding  stage.  The  response  summary 
combines  those  program  responses  with  information  received 
for  additional  interview  subjects  and  respondents  who 
answered  from  supervisory  positions  reflecting  upon 
multiple  programs. 
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RELIABILITY  OF  FIELDED  SYSTEMS 

YES 

NO 

# 

%  # 

o 

o 

Conditional  Materiel  Release  (CMR) 

Was  the  system  initially  fielded  with  a  CMR  due  to 
reliability  shortfalls? 

3 

Is  the  CMR  still  in  effect? 

N/A 

Collection  of  Field  Reliability  Data 

Reliability  information  is  obtained  from  Depot,  Contractor 
Logistics  Support  (CLS)  records,  or  other  means  (e.g. 
Production  Quality  Deficiency  Reports  PQDRs) 

4 

Warranty  collection  data  provides  information  on  reliability 
performance 

A  formal  collection  system  does  not  exist 

Status  of  Reliability  in  the  Field 

System  performance  meets/exceeds  ORD  requirement 

3 

System  performance  is  less  than  ORD  requirement 

Do  not  know  (due  to  lack  of  data,  or  too  early) 

Cost  Effective  Reliability  Improvements 

Has  collection  of  reliability  failure  data  in  the  field  led 
to  any  cost  effective  improvements? 

3 

Too  early  in  program  to  tell 

Reliability  Improvement  Program 

Is  there  a  formal  reliability  improvement  program? 

2 

3 

Reliability  Centered  Maintenance  (RCM) 

Is  there  a  formal  RCM  program? 

4 

1 

Table  16.  Reliability  of  Fielded  Systems 


Response  Summary:  Fielded  systems  often  are  seen  as 

the  benchmark  of  success,  but  the  hard  work  has  just  begun. 


59 


Ideally,  the  processes  used  to  assess  and  enhance  system 
reliability  throughout  the  acquisition  cycle  will  pay  off. 
Compromises  along  the  way  however,  have  invariably  taken  a 
toll  on  the  optimization  of  reliability. 

Experience  seems  to  have  demonstrated  that  the 
reliability  activities  used  in  the  design,  development  and 
production  of  a  system  have  been  geared  to  meet  a 
specification  requirement  that  may  not  reflect  the  actual 
operators  use  of  the  system.  Programs  sometimes  complain 
that  the  user  does  not  operate  a  system  as  it  was  designed, 
while  operators  retort  that  the  system  was  not  designed  to 
be  operated  in  the  manner  they  require. 

Disconnects  between  the  program  office  and  the 
warfighter  seem  to  be  traced  to  communication  failures. 
Either,  there  is  an  actual  problem  with  the  weapon  system, 
or  with  its  associated  training  plan.  Both  of  these  items 
fall  within  the  purview  and  responsibilities  of  the  PM. 

There  is  little  consistency  between  the  programs. 
Some  systems  state  reliability  in  different  ways.  Here  are 
three  weapon  system  examples:  1)  The  weapons  will  be  at 
least  XX%  reliable  when  removed  from  its  container  at  any 
time  during  its  life;  2)  The  weapons  will  be  least  XX% 
reliable  when  removed  from  it  container  after  five  years  in 
storage  or  3)  Weapons  must  be  at  least  XX%  reliable  when 
tested  after  three  years  and  may  degrade  no  more  the  Y%  for 
each  additional  year.  Other  systems  state  the  requirement 
in  terms  of  Storage  MTBF,  Captive  Carry  MTBF  or  both. 

The  collection  of  field  data  is  a  crucial  element  in 
reliability  support.  Aircraft  systems  generally  provide 
maintenance  information  through  automated  information 
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systems.  Naval  Aviation  Logistics  Command  Management 
Information  System  (NALCOMIS)  collects  and  provides 
automation  for  organizational  and  intermediate  level 
maintenance  activities.  NALCOMIS  is  generally  used  as  a 
day-to-day  management  tool  and  holds  maintenance  related 
information  including  historical  logs  and  records.  [Ref. 
29] 

The  need  for  improved  data  analysis,  resulting  from 
the  growth  in  sophistication  and  complexity  of  weapon 
systems,  has  led  to  the  development  of  the  Naval  Aviation 
Logistics  Data  Analysis  (NALDA)  System.  [Ref.  36]  However, 
reliability  practitioners  feel  that  this  system  provides  a 
wealth  of  data  but  does  not  directly  correlate  to 
reliability  uses.  NAVAIR  does  have  at  least  three 
activities,  in  varying  stages  of  implementation,  focused  on 
in-service  management  that  incorporates  reliability.  The 
three  activities  are  Reliability  Centered  Maintenance 
(RCM) ,  Sustained  Maintenance  Planning  (SMP) ,  and 
Reliability  Improvement  Teams  located  at  the  depots. 

RCM  is  an  analytical  process  used  to  determine 
preventive  maintenance  (PM)  requirements.  Used  as  a  tactic, 
RCM  is  a  strategy  that  is  function  oriented  seeking  to 
preserve  system  functions  through  maintenance  at  failure, 
preventive  maintenance,  predictive  maintenance  and 
detective  maintenance.  The  basic  fundamental  of  RCM  is  the 
root  cause  analysis  of  why  a  failure  occurred.  The  root 
cause  needs  to  be  determined  so  that  an  appropriate 
solution  can  be  invoked.  [Ref.  21] 

SMP  and  RIT  both  stem  from  the  Business  Process  Review 
(BPR)  initiative.  SMP  aims  not  only  at  identifying 
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problems  but  also  attempts  to  examine  proactive 
opportunities  to  infuse  new  technologies,  obsolescence 
avoidance,  and  maintenance  improvements  encompassing 
additional  perspectives  beyond  reliability.  SMP 

implementation  has  been  program  specific  vice  a  centralized 
NAVAIR  process. 

The  reliability  improvement  teams  look  to  improve 
component  reliability  and  lower  life  cycle  costs  through 
the  identification  of  high  cost  and/or  low  reliability 
items.  Once  the  components  are  identified,  processes  and 
results  from  maintenance  and  repair  are  studied  to  identify 
improvements.  The  intent  is  to  complete  a  full  loop  from 
problem  identification,  fix  implementation,  through 
solution  verification.  Their  reliability  analysis  model  is 
currently  in  its  initial  phase  with  a  low-tech  approach. 
Figure  8  depicts  the  three-phased  development  of  the 
program. 

Supporting  Comments : 

•  TAFT  was  painful,  but  success  was  in  the 
fielding . 

•  Most  systems  do  employ  RCM,  but  "hands-on"  data 
collection  and  scrubbing  is  very  inefficient  and 
labor  intensive. 

•  No  extensive  reliability  data  collection  system 
available.  No  S/N  tie  to  failure  data  for  0/I/D 
level  life  cycle  tracking  -  NEED  ONE  !  We're 
"data  rich  -  info  poor" . 

•  Need  to  identify  reliability  data  collection 
goals  (time  on  wing,  rework  requirements,  MP 
support,  RCM/PM  reqts,  etc.)  and  develop  the  data 
collection  system  to  support  those  goals. 

•  While  real  time  supply  support  is  certainly  a 
plus,  have  concerns  with  TLS  contracts  in  regards 
to  prime  "preaching"  reliability  improvements. 
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stating  irrelevant  metrics,  and  not  producing 
improvements . 

•  RCM  for  weapons  is  still  a  work  in  progress. 
Weapons  reliability  is  driven  by  electronics 
random  failure;  this  failure  mechanism  is  not 
suited  to  RCM. 


NAVAL  AVUTKM  SYSTEyS 


Reliability  Analysis  Model 


Select  Analyze  Fix  Measure 


PHASE  THREE 
•INSTITUTIONALIZED 
CAPABILITIES 
•PERFORMANCE  BASED 
INDUSTRIAL  FOCUS 
•FORMAL  LIFE  CYCLE 
MODELING 

PHASE  TWO 
•EXPANDED  FOCUS  TO 
DESIGN  /  PERFORMANCE 
•EXPANDED  KNOWLEDGE 
OF  FAILURE  MODE  / 
MECHANISM 
•BEGIN  FORMAL  MODELING 

PHASE  ONE 

•TARGET  TOP  COST  DRIVERS 
•REACH  INHERENT 
RELIABILITY 
•INDUSTRIAL  PROCESS 
FOCUS 
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Figure  7 . 


Business  Process  Review  Reliability  Analysis  Model 

[From  Ref. 4] 


H.  AFFECTS  OF  ACQUISITION  REFORM 

Acquisition  reform  is  a  leadership-sponsored  movement 
throughout  the  Department  of  Defense  endeavoring  to  make 
the  acquisition  process  more  effective,  efficient  and 
productive.  The  reform  initiative  is  intended  to 
streamline  requirements  and  reduce  the  system's  cycle  time. 
Moves  toward  the  use  of  commercial  practices  and  private 
enterprise  are  hallmarks  of  Acquisition  reform. 
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On  June  29,  1994,  Department  of  Defense  Secretary 
William  Perry  mandated  the  use  of  acquisition  reform 
throughout  DoD  with  the  issuance  of  his  memorandum 
entitled,  "Specifications  and  Standards--A  New  Way  of  Doing 
Business."  The  implementation  of  streamlining  measures  and 
personnel  reductions  had  effects  throughout  the  acquisition 
community.  The  researcher  sought  to  determine  the 
perceived  effects  of  reform  on  the  engineering  discipline 
of  reliability  and  its  management.  [Ref.  23} 


Survey  questionnaire  participants  were  aslced: 

In  your  opinion,  has  acquisition  streamlining  (e.g. 
performance  specifications,  use  of  COTS,  etc...)  and/or  the 
continued  trend  of  government  downsizing  contributed  either 
directly  or  indirectly  towards  reliability  shortfalls 
experienced  by  programs? 

Additionally,  two  follow-up  questions  are  included  in 
Table  17  and  the  response  summary  below; 

a)  If  COTS/NDI  components  were/are  utilized  in  the 
design  of  your  system,  did  the  COTS  components  realize  the 
reliability  performance  claims  of  the  OEM?  b)  Given  the 
realities  of  streamlining  and  downsizing,  do  you  believe 
the  Navy  reliability  community  has  adequately  compensated 
with  alternative  policies,  processes  and  tools? 
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In  your  opinion,  has  the  move  towards  performance-based 
specifications,  the  increased  use  of  COTS,  and/or  the 
continued  trend  of  Government  downsizing  had  any 
negative  effects  on  reliability  of  systems? 

Program 

Responses 

%  of 

Programs 

Yes,  due  to  performance  based  specifications. 

4 

31% 

Yes,  due  to  downsizing  the  workforce. 

Yes,  due  to  both  acquisition  streamlining  and  downsizing 

5 

38% 

No 

2 

15% 

No  comment 

4 

31% 

COTS/NDI  components  do  not  live  up  to  OEM  claims 

3 

23% 

Table  17.  Effects  of  Acquisition  Reform  Initiatives 


Response  Summary: 

Respondents  and  interviewees  generally  expressed 
concern  that  although  acquisition  reform  has  benefited  some 
areas,  reliability  practices  and  reliability  management 
have  specifically  been  hampered.  The  combination  of  a 
reduction  in  the  reliability  workforce  and  increased  use  of 
commercial  of  the  shelf  (COTS)  items  continually  were 
highlighted  as  problem  areas. 

The  reliability  workforce  has  declined  while  the 
increased  use  of  commercial  items  and  performance 
specifications  actually  has  increased  the  total  workload. 
COTS  usage  has  proven  to  be  more  work  than  it  appears 
because  COTS  equipment  has  not  been  designed  to  operate  in 
the  warfighters  environment. 


Additional  Paraphrased  Comments : 

•  The  perception  by  program  offices  is  that  all  you 
have  to  do  with  COTS  is  install  it— no  need  to 
test . 
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•  COTS  has  major  limitations  and  upper  management 
does  not  understand  this  or  even  want  to  discuss 
it . 

•  Programs  have  found  that  on  COTS  where  they  did 
HALT  testing,  they  improved  the  system  and  expect 
it  these  systems  to  work  in  our  environment. 

•  PMs  need  to  determine  if  COTS  items  will  survive 
in  your  operating  environment. 

•  Contractors  have  stated  that  are  going  to 
commercial  components  and  that  we  have  to  change 
our  environmental  requirements,  as  their  systems 
will  no  longer  meet  the  existing  standards. 

•  When  some  PMAs  receive  a  request  to  use  COTS  or 
commercial  components  in  a  design,  they  require 
HALT. 

•  Some  programs  use  COTS  but  always  have  to  harden 
them  to  work  in  our  environs— so  are  they  still 
COTS? 

•  The  reduced  use  of  specs  and  standards  has 
hampered  our  effectiveness. 

•  Streamlining  means  the  use  of  COTS,  less  people 
to  track  R&M,  less  people  to  follow  up  and  fix 
issues . 

•  Reliability  personnel  cuts  led  to  the  current 

method  of  monitoring  results  from  the  Fleet, 
which  is  too  long  to  provide  any  meaningful 
feedback  to  the  manufacturer  to  help  in 

reliability  growth.  Systems  are  out  of  warranty 
before  they  are  evaluated  for  reliability. . 

•  Ignorance  is  not  an  acceptable  solution  to 
inadequately  funded/resourced  effort.  Additional 
risk  not  only  kills  programs,  it  kills  people  and 
combat  effectiveness. 


Paraphrased _ Respondent _ Suggestions  :  Survey 

questionnaire  respondents  were  asked  if  they  had 
suggestions  for  improvement  to  acquisition  reform. 
Selected  comments  are  listed  below: 
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•  Mandate  the  use  of  life-cycle-costs  as  the  basis 
for  all  design  trade-offs.  Currently  the  unit 
production  cost  is  used,  therefore  reliability 
and  life-cycle-costs  are  always  sacrificed  to 
produce  a  lower  unit  cost.  The  unit  cost  is 
generally  the  metric  which  determines  the  success 
of  the  program  for  the  Milestone  Decision 
Authority  (MDA) . 

•  Need  to  standardize  modeling  tools  used  within 
NAVAIR. 

•  Working  with  USAF;  their  standardization  of 
processes  are  documented  better. 

•  Do  not  use  COTS.  Use  what  is  best  to  meet 
reliability  goals  of  the  program. 

•  Streamlining  and  downsizing,  to  be  effective, 

requires  all  parties  to  assume  greater 

responsibility.  In  practice  we  have  spread  so 
thin  and  not  held  anyone  accountable.  We  need  to 
empower  (and  reward)  people  to  accept  the 
additional  responsibility  that  has  been  thrust 
upon  them. 

•  Fund  and  resource  to  adequate  levels. 

•  R&M  and  supportability  are  always  compromised 

when  executing  programs.  These  activities  need 

to  be  funded  and  their  schedules  need  to  be 
shielded  from  compression  that  would  make  them 
meaningless.  We  will  pay  up-front  or  the  fleet 
WILL  pay  once  it  gets  fielded. 


All  surveyed  programs  were  asked  to  identify  which 
reliability  related  commercial  practices  were  employed  in 
their  program.  Table  18  presents  their  responses. 
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What  Types  of  Commercial  Reliability 
Assurance  Practices  Do  You  Employ  in 
Your  Program? 

Program 

Responses 

0, 

o 

Physics  of  Failure  (PoF) 

3 

23% 

Predictive  Models 

3 

23% 

Prognostics/Life  Consumption 

Monitoring 

2 

15% 

Identification  and  Mitigation  of 
Failure  Modes  (FMECA) 

6 

46% 

Accelerated  Life  Testing  (e.g.  HALT) 

4 

31% 

Reliability  Growth  Testing 

2 

15% 

Reliability-Driven  Parts 
Selection/Control 

3 

23% 

Other 

3 

23% 

Do  Not  Employ  any  Commercial  Practices 

3 

23% 

Table  18 


Commercial  Reliability  Practices 


Response  Summary: 

Commercial  practices  are  widely  used  in  Naval  aviation 
programs.  Programs  tailor  the  practices  employed  to  fit 
their  reliability  goals  and  more  importantly  their  budget. 

Additional  Paraphrased  Comments: 

•  Need  to  benchmark  some  commercial  systems 
(believe  Northwest  has  an  efficient  data 
collection/analysis  system). 

•  Our  practices  are  reactive  programatics ,  not  a 
deliberate  effort  to  effectively  mitigate  risks. 

I .  CHAPTER  SUMMARY 

This  chapter  presented  the  methodology  used  and  the 
data  gathered  relating  to  the  task  of  reliability 
management.  Acquisition  workforce  professionals  directly 
contributed  through  participation  in  either  interviews 
and/or  completion  of  a  survey  styled  questionnaire.  The 
information  provided  reflects  the  experience  and 
perceptions  of  the  experts  who  know  the  issues  involved 
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with  reliability  management.  The  data  was  organized  in 
four  categories:  1)  Reliability  Management  Environment;  2) 
Reliability  Processes  and  Tools;  3)  Reliability  of  Fielded 


Systems;  and  4)  Affects  of  Acquisition  Reform. 

Chapter  IV  provides  an  organized  analysis  of 
presented  in  this  chapter,  focusing  on  the 
available  to  the  PM  to  optimize  reliability. 


the  data 
methods 
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IV.  J^ALYSIS  OF  RELIABILITY  MANAGEMENT 


A.  INTRODUCTION 

This  chapter  discusses  the  results  of  research  and  the 
data  presented  in  Chapters  II  and  III.  The  focus  of 
analysis  is  on  the  primary  thesis  question: 

"What  strategies  should  Program  Managers  implement  to 
optimize  reliability  in  their  weapon  system?" 

The  qualitative  analysis  presented  will  follow  the 
format  of  the  data  presented  in  the  previous  chapter, 
including  the  following  four  themed  categories:  1) 
Reliability  Management  Environment;  2)  Reliability  Tools, 
Techniques  and  Processes;  3)  Reliability  of  Fielded 
Systems;  4)  Affects  of  Acquisition  Reform. 

B.  RELIABILITY  MANAGEMENT  ENVIRONMENT 

1 .  Reliability  Responsibility 

The  Program  Manager  is  ultimately  responsible  for 
reliability,  and  the  reliability  results  achieved  will 
reflect  the  PMs  attention  to  the  issue.  Tailoring  may 
account  for  some  differences  between  programs  but  if 
individuals  within  the  same  program  cannot  identify  the 
body  responsible  for  reliability  management  there  is  a 
significant  lack  of  leadership  attention  to  the  subject. 

Although  it  may  seem  logical  that  an  IPT  created  to 
deal  with  program  reliability  issues  would  have  primary 
responsibility  of  its  management,  it  also  can  be  perceived 
as  a  stovepipe  or  constricting  process.  The  programs  that 
assigned  the  reliability  members  to  each  design  team  claim 
to  have  done  so  with  purpose.  Through  the  dissemination  of 
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reliability  expertise  these  programs  aim  to  catch 
problematic  issues  before  they  are  instituted.  This 
forward  thinking  approach  displays  an  attitude  toward 
reliability  that  shifts  its  management  from  supervisory  to 
participatory . 

Even  with  logistical  and  reliability  personnel 
disseminated  throughout  a  program  there  should  still  be  a 
recognizable  champion  for  reliability  issues.  On  programs 
where  there  is  not  a  consistent  designated  reliability 
authority  there  may  be  a  lack  of  focus  to  the  subject. 

2 .  Reliability  Dociamentation 

The  data  shows  that  documenting  reliability  on  a 
program  level  is  accomplished  most  often  through  the  use  of 
a  dedicated  Reliability  Program  Plan  (RPP) .  Although 
programs  have  developed  a  document  detailing  the  activities 
and  responsibilities  surrounding  the  management  of 
reliability,  it  should  be  noted  that  none  of  the  programs 
provided  their  RPP  to  the  researcher.  Therefore  a  thorough 
analysis  of  the  contents  of  these  plans  was  not  possible. 
Review  of  these  documents  is  warranted  and  is  listed  in 
Chapter  V  as  an  area  for  further  research.  It  is  critical 
that  PMAs  have  reliability  criteria  that  is  clear  and  can 
be  rigorously  enforced. 

3.  Reliability  Resources 

Acquisition  management  is  accomplished  in  a  fiscal 
reality  where  resources  are  limited  and  true  costs  are  not 
articulated.  As  such,  any  analysis  done  must  appreciate 
the  gray  area  of  the  political  arena  in  which  PMs  operate. 
It  is  not  that  limited  resources  applied  to  reliability 
cannot  be  explained  or  appreciated,  but  the  acquisition 
community  and  its  leadership  must  knowingly  acknowledge 
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that  money  not  spent  on  reliability  optimization  during 
development  will  be  spent  many  times  over  during  operations 
and  support . 

This  fiscal  death  spiral  is  where  the  services 
currently  find  themselves  attempting  to  maintain  a  legacy 
system  laden  defense  force.  Although  current  budgeting 
events  surrounding  the  War  on  Terrorism  are  likely  to  yield 
positive  increases  in  recapitalization  funding,  if  those 
recapitalization  efforts  do  not  reflect  reliability 
improvement  measures  there  will  have  been  no  learning  from 
the  fleets  present  condition  and  existing  systems  will 
continue  their  downward  slide. 

The  competition  for  funding  that  pits  reliability 
improvements  against  performance  enhancements  is 
foreboding.  Often  the  system  performance  improvements  are 
made  not  only  at  the  expense  of  possible  reliability 
enhancements,  but  actually  further  degrade  the  existing 
reliability.  The  lack  of  emphasis  on  reliability  resources 
is  counterproductive  to  any  life  cycle  concern. 

4 .  Reliability  Regulations  and  Policies 

The  current  state  of  regulatory  documents  and  policies 
seems  insufficient  for  use  as  constructive  managerial 
tools.  Although  the  advent  of  the  Spec  and  Standards 
Reform  does  allow  great  latitude  for  PMs  to  tailor  and 
apply  commercial  standards  to  suit  their  needs,  the 
commercial  standards  for  reliability  do  not  reflect  the 
reliability  needs  of  DoD.  In  practice,  programs  continue 
to  use  the  cancelled  specs  and  standards  through  a  cut  and 
paste  mentality  placing  the  MIL-SPEC  information  in  the 
Statement  of  Work. 


73 


Some  might  contend  that  this  approach  is  perfectly 
aligned  with  the  reform  movement,  which  allows  the  use  of 
Government  standards  where  required.  However,  the 
Government  reliability  standards  are  no  longer  reviewed  or 
maintained  on  a  regular  basis.  Despite  the  efforts  of  the 
Reliability  Research  Center  no  commercial  standard  exists 
for  reliability  engineering  or  management. 

The  pending  rewrite  of  the  SECNAVINST  5000. 2C  has 
reliability  specific  guidance  that  should  be  a  useful  aid 
to  PMs .  A  consistent  recognition  of,  and  focus  upon,  the 
importance  of  reliability  management  will  greatly  benefit 
the  life  cycle  cost  burden  under  which  our  current  systems 
suffer.  The  acquisition  workforce  personnel  responsible 
for  reliability,  including  the  logisticians,  engineers  and 
testers,  require  enforceable  criteria  and  clearly 
delineated  reliability  management  fundamental  procedures. 

C.  RELIABILITY  TOOLS,  TECHNIQUES  AND  PROCESSES 

1 .  Requirements  Generation 

The  data  shows  that  PMAs  are  involved  in  the 
reliability  requirement  generation  in  most  cases.  PMA 
involvement  implies  an  opportunity  for  PMs  to  have  a 
positive  influence  on  the  incorporation  of  realistic 
reliability  requirements.  The  extent  to  which  this 
positive  influence  is  effective  is  reflective  of 
reliability  and  logistical  experts  level  of  involvement  and 
their  understanding  of  the  criticality  of  reliability. 

It  is  essential  that  the  logisticians  and  reliability 
personnel  be  involved  in  this  early  stage  of  the  program 
development.  The  issue  of  reliability  truly  should  be 


multi-disciplinary  and  cannot  be  relegated  to  the 
engineering  disciplines.  Reliability  is  a  fundamental 
component  of  a  systems  development  and  requires  input  from 
all  sources.  A  reliability  oversight  during  requirements 
generation  can  take  years  to  fix  and  is  difficult  to 
incorporate  into  an  existing  program. 

2 .  Design  for  Reliability 

Design  emphasis  must  reflect  the  required  reliability 
and  robustness  required  by  the  warfighter' s  operational 
environment.  The  PM,  using  the  Systems  Engineering  Process 
with  strong  input  from  logistics  management,  is  responsible 
for  taking  validated  user  requirements  and  turning  them 
into  design  criteria.  A  balance  must  be  struck  between 
cost,  schedule,  performance,  risk  and  life  cycle  needs. 
Reliability  certainly  falls  within  this  purview. 

It  is  up  to  the  PMA  to  exploit  opportunities  to  design 
for  reliability,  manufacturability  and  overall 
producibility .  Unlike  the  PM,  the  warfighters  and  user 
representatives  are  not  likely  familiar  with  these 
concerns.  They  will  concentrate  on  performance  figures  and 
production  numbers.  Concepts  like  simplicity  of  design 
must  be  presented  in  terms  of  increased  reliability, 
maintainability  and  operational  availability.  Each  of  the 
system' s  parts  is  in  effect  a  building  block  with  inherent 
reliability . 

Each  of  the  participating  programs  reported  the  use  of 
design  tools  that  address  reliability.  The  effectiveness 
of  these  tools  is  of  course  to  be  determined  over  time  with 
each  systems  usage.  But  the  acknowledgement  of  the  need 
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for  reliability  enhancing  tools  in  system  design  is  a  good 
thing . 

3.  Contracting  for  Reliability 

If  it  is  important  put  it  in  the  contract.  To 
influence  reliability  design  requirements,  the  contract 
must  articulate  their  importance.  Motivation  articulated 
through  incentives  is  an  effective  strategy  encouraging 
contractors  to  apply  their  best  efforts  on  the  issue. 

Contractual  reliability  incentive  usage  within  the 
participating  programs  is  encouraging.  Although  less  than 
half  of  the  programs  report  the  usage  of  incentives,  the 
programs  that  have  implemented  incentives  appear  to  have 
learned  the  lesson.  Their  usage  of  award  fees  to  focus  the 
contractor' s  efforts  on  reliability  improvement  is  a 
positive  step  toward  optimization  of  reliability. 

Overall,  PMAs  must  realize  that  realize  that  many 
areas  affect  reliability  and  that  participation  with  the 
contractor  during  system  design  is  essential.  To  influence 
design— put  it  in  the  contract  and  monitor  contractor 
compliance.  Only  through  vigorous  involvement  can  the  PMA 
ensure  and  enforce  the  contractors  focus  on  reliability. 

4.  Reliability  Test  and  Evaluation  Activities 

There  is  a  consistent  cultural  conflict  between  the 
test  community  and  the  PM  shop.  However,  test  and 
evaluation  (T&E)  is  an  integral  part  of  every  program.  T&E 
is  undertaken  both  to  aid  in  the  development  of  a  system 
and  to  verify  that  a  given  product  meets  the  standards  as 
stated  by  the  customer.  The  respondent  and  interview 
comments  indicate  that  PMAs  generally  think  that  some 
testers  test  to  unrealistic  mature  reliability  thresholds 
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before  a  system  is  allowed  to  mature,  and  do  not  have  an 
appreciation  for  when  the  user  needs  are  met. 

A  defined  reliability  growth  program  is  a  helpful  tool 
to  help  articulate  the  PM  point  of  view.  A  realistic 
growth  strategy  that  develops  as  the  product  itself  grows 
is  prudent  and  would  be  useful  to  testers  and  decision 
makers  alike.  The  growth  of  system  reliability  cannot  be 
taken  for  granted  and  must  be  consistently  monitored  and 
tested  to  ensure  compliance  with  the  user's  needs. 
Additional  tests  may  be  required  either  to  verify  product 
improvements  or  simply  to  obtain  a  reasonable  confidence 
level  in  the  system's  results. 

Although  there  is  PMA  appreciation  for  the  risk  that 
not  meeting  reliability  requirements  poses  to  a  program, 
there  is  not  a  consistent  pattern  of  risk  management 
applied  to  reliability  management.  Tailoring  may  again 
rightfully  address  why  there  is  not  a  uniform  set  of 
procedures  used  to  test  reliability,  but  does  not  address 
the  perception  that  the  lack  of  reliability  testing  is 
knowingly  accepted  because  of  an  unwillingness  of  PMs  to 
receive  bad  news  or  accept  schedule  required  adjustments. 

PM  use  of  realistic  reliability  entrance  criteria  to 
stages  of  testing  including  OPEVAL,  and  having  clearly 
articulated  reliability  KPPs  are  effective  management  tools 
and  would  help  establish  required  reliability  thresholds 
for  which  programs  could  strive.  A  tailored  selection  of 
environmental  and  accelerated  tests  combined  with  an 
effective  failure  reporting  system  that  incorporates 
corrective  actions  could  be  used  to  demonstrate  a  program' s 
seriousness  when  addressing  reliability.  Experience 
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continues  to  demonstrate  that  it  does  little  good  to 
complete  testing  but  fail  to  act  upon  the  knowledge  gained. 

A  null  hypothesis  must  be  applied  to  vendor  and 
program  claims  of  MTBF  levels.  The  testers  and  PMAs  must 
demand  proof  that  a  product  will  indeed  perform  in  the  DoD 
application  to  the  claimed  level.  This  testing  must  be 
accomplished  with  a  level  of  confidence  that  allows 
decision  makers  to  make  informed  judgments  about  a  system's 
production  and  support  requirements. 

D.  RELIABILITY  OF  FIELDED  SYSTEMS 

Success  is  too  often  believed  to  have  occurred  upon 
reaching  a  full  rate  production  decision  or  upon  initial 
fielding.  Even  when  the  war  fighting  units  have  been  fully 
outfitted,  program  management  work  remains.  Weapon  systems 
and  their  logistic  support  structure  must  be  carefully 
monitored,  measured,  and  adjusted  as  appropriate. 

Collection  of  field  reliability  performance  data  is 
not  uniform  and  hard  to  accomplish.  Warfighters  are  more 
concerned  with  performance  measures  than  with  logistical 
concerns,  until  there  is  a  problem.  Once  a  problematic 
situation  has  risen  to  the  attention  of  the  fleet  user  the 
PMs  job  has  gotten  harder.  Programs  that  rely  upon 
reactive  means  for  identifying  and  correcting  problems  will 
terminally  be  on  the  defensive. 

Proactive  measures  pay  dividends  in  terms  of  the 
fleet's  Operational  Availability  (Ao) .  However,  those 
proactive  programs  do  not  come  without  a  cost.  Resources 
in  the  form  of  time,  money  and  technology  must  be  spent  to 
collect  in-service  equipment  data.  Additional  resources 
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must  then  be  applied  to  identify  and  implement  corrective 
actions  as  required.  Unfortunately,  logistical  support  of 
fielded  systems  is  funded  with  operational  support  dollars 
and  often  insufficient  to  permit  reliability  improvements. 

Centrally  collected  data  needs  to  be  formulated  in  a 
manner  that  usefully  reflects  system  reliability.  This 
data  should  be  evaluated  and  acted  upon.  The  addition  of  a 
specific  Flag-level  reliability  review  would  highlight  the 
importance  of  reliability,  and  could  improve  the  situation 
if  the  data  is  formulated  as  value  added  information.  If 
the  acquisition  and  operational  leadership  is  truly 
concerned  with  reliability  as  a  life  cycle  cost  driver 
their  actions  would  speak  volumes  versus  toothless 
statements  placed  within  strategic  plans. 

E.  EFFECT  OF  ACQUISITION  REFORM 

Three  by-products  of  acquisition  reform  were  cited  as 
having  negative  effects  on  reliability  of  systems.  First, 
the  implementation  of  performance  based  specifications 
replacing  the  Government  standards  that  dictated 
reliability  practices;  Second,  the  reduction  in  the 
reliability  workforce  combined  with  streamlining;  Third, 
the  misuse  or  misunderstanding  of  commercial-off-the-shelf 
(COTS)  items. 

The  shift  away  from  Government  documents  towards 
commercial  practices  and  industry  standards  certainly  has 
had  merit  in  some  areas.  However,  the  general  perception 
from  the  reliability  duty  experts  was  that  no  industry 
standard  exists  and  that  there  is  yet  to  be  any  gain  from 
the  use  of  commercial  practices.  There  are  encouraging 
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areas,  such  as  the  increased  application  of  HALT,  but  no 
consistent  standards  from  which  to  measure. 

The  significance  of  the  workforce  reduction  is 
amplified  when  it  is  combined  with  the  increased  use  of 
COTS.  COTS  items  are  too  often  thought  of  as  a  quick  and 
easy  bolt  on  fix  that  requires  little  or  no  testing.  In 
reality,  COTS  items  have  not  been  designed  for  the 
warfighting  environment  and  require  reliability  testing  to 
ensure  their  success  in  the  DoD  application.  If  a  COTS 
item  is  then  found  to  be  lacking,  appropriate  modifications 
must  be  made  in  order  to  meet  the  user's  needs. 

The  mentality  toward  COTS  that  "it  already  works, 
that's  why  we  are  buying  it"  is  ill  informed  and  unwise. 
The  commercial  environment  seldom  demands  the  life 
expectancy  too  which  DoD  has  become  accustomed.  It  has 
become  common  for  DoD  systems  to  be  stretched  beyond  their 
initial  service  lives,  which  generally  were  longer  than 
most  businesses  keep  their  capital  equipment.  COTS  usage 
can  provide  substantial  benefits  to  DoD  acquisition,  but 
the  COTS  application  must  fit  the  requirement,  the  reverse 
is  unacceptable. 


F .  CHAPTER  SUMMARY 

Program  and  reliability  management  personnel  are 
charged  with  the  responsibility  of  fielding  reliable 
equipment  to  the  nation' s  warfighters  and  should  be  held 
accountable  for  that  mission's  accomplishment.  This 

chapter  provided  an  analysis  of  the  reliability  related 
data  presented  in  the  previous  chapters.  Chapter  V  will 
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present  synopsized  answers  to  the  research  question  and 
selected  conclusions  and  recommendations. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  INTRODUCTION 

The  goal  of  this  thesis  was  to  provide  strategic 
planners  with  insight  collected  from  the  experience  of 
acquisition  workforce  professionals  familiar  with  the 
issues  relating  to  reliability  management.  Application  of 
the  cumulative  empirical  evidence  of  these  professionals 
permits  a  forward  leaning  proactive  approach  to  the 
optimization  of  reliability  within  Department  of  Defense 
(DoD)  acquisition  programs. 

B.  ANSWERS  TO  RESEARCH  QUESTIONS 

Synopsized  answers  to  the  research  question  are 
provided  here.  Additional  insight  and  information  have 
been  provided  in  Chapters  II  and  IV. 

1 .  What  Strategies  Should  Program  Managers  Implement 
to  Optimize  Reliability  in  Their  Weapon  System? 

Communicate  and  champion  the  optimization  of 
reliability.  It  is  human  nature  to  cultivate  the  areas 
where  one  is  held  accountable. 

Ensure  requirements  are  grounded  in  reality  and 
articulate  reliability  concerns  early  and  often. 

Embrace  testing  as  a  value  added  dynamic.  Use  all 
test  and  evaluation  opportunities  as  learning  and 
verification  tools  justifying  required  resources.  Apply 
test  results  to  improve  the  system.  Problems  identified 
through  testing  (even  if  they  are  only  a  matter  of 
perception)  already  exist  and  would  have  been  found 
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eventually.  Treat  those  identified  problems  as  knowledge 
gained . 

Assume  the  null  hypothesis  that  the  contractors' 
reliability  claims  are  flawed  until  proven  otherwise.  This 
does  not  predetermine  an  adversarial  relationship  with  a 
contractor  but  clearly  articulates  the  Government's 
position  form  the  start. 

Optimize  trade-offs  with  respect  to  reliability  and 
fully  appreciate  the  cause  and  effect  relationship 
decisions  today  will  have  on  the  fleet  tomorrow. 

Build  a  team  of  professionals  skilled  and  educated  on 
issues  of  reliability  so  that  they  may  permeate  the  program 
articulating  and  implementing  a  sound  acquisition  strategy 
reflective  of  life  cycle  concerns. 

2 .  What  is  Reliability  and  What  is  its  Significance 
within  Acquisition  Management? 

Reliability  is  the  probability  an  item  will  perform 
its  intended  functions  for  a  specified  period  under  stated 
conditions . 

Reliability  is  the  fundamental  building  block  that 
supports  the  warfighter.  It  directly  supports  the 
operational  availability  of  systems  and  is  the  significant 
driver  of  life  cycle  costs.  System  logistical  support  is 
based  upon  the  expected  component  and  total  system 
reliabilities.  Poor  reliability  or  inaccurate  estimates  of 
reliability  significantly  increase  life  cycle  costs  and 
affects  every  aspect  of  logistics  elements. 
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3 .  What  Policies  and  Regulations  Governing 
Reliability  Management  are  Available  to  Program 
Managers  (PM) ? 

DoD  5000. 2-R  is  the  overarching  regulatory  document 
requiring  Program  Managers  to  establish  and  execute 
reliability  activities  and  measurable  requirements  within 
their  acquisition  programs.  However,  since  the  issuance  of 
the  Secretary  Perry' s  memo  on  specifications  and  standards 
reform  most  of  the  guidance  on  reliability  has  purposely 
become  vague.  Therefore  there  are  no  governing  documents 
that  provide  procedural  guidance  to  the  Program  Managers . 

In  their  absence  the  managers,  and  engineers 
responsible  for  reliability,  use  a  number  of  discretionary 
guides  from  the  Defense  Systems  Management  College, 
'toolkits'  from  the  Reliability  Analysis  Center,  and  the 
cancelled  reliability  military  specs  and  handbooks.  The 
draft  copy  of  the  SECNAVINST  5000. 2C  reportedly  makes  the 
most  significant  advance  in  the  area  of  R&M  since  the 
advent  of  acquisition  reform.  Hopefully  its  release  will 
regain  some  of  the  reliability  ground  lost. 

4 .  What  Significant  Factors  Contribute  to  Weapon 
System  Reliability? 

First  and  foremost  system  reliability  stems  from 
command  attention.  In  terms  of  product  development,  the 
design  process  and  requirements  generation  phases  provide 
the  critical  reliability  foundation.  Following  a  credible 
design,  the  development  and  manufacturing  processes  must  be 
formulated  and  adjusted  to  enhance  the  inherent  reliability 
of  the  product  being  produced.  A  proactive  reliability 
plan  combined  with  applied  lessons  learned  from  testing 
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iterations  will  significantly  enhance  the  opportunity  for  a 
reliable  system  success. 

Additionally,  the  use  of  contractual  incentives,  KPPs, 
and  entrance  criteria  linked  to  reliability  thresholds  will 
help  raise  the  level  of  effort  applied  to  the  life  cycle 
cost  driver;  reliability. 

5.  What  Strategies  are  Currently  Used  to  Monitor 
Reliability  within  an  Acquisition  Program? 

The  research  identified  no  uniform  strategy 
specifically  aimed  at  monitoring  reliability.  Programs  use 
a  variety  of  methods  to  track  reliability  including 
reliability  program  plans,  reliability  integrated  product 
teams,  major  program  reviews,  developmental  and  operational 
test  reports.  Additionally,  some  programs  have 
incorporated  semi-annual  reliability  meetings  or  reports 
and  encourage  three  to  six  month  reliability  reviews  on 
field  equipment. 


6.  How  can  the  Program  Office  Mitigate  Risk 

Associated  with  Reliability  Throughout  the 
Acquisition  Life  Cycle? 

Understand  and  articulate  the  significance  of 
reliability  with  respect  to  life  cycle  costs  and  its  affect 
on  the  warfighter. 

Review  policies,  actions  and  decisions  through  the 
eyes  of  the  warfighter  with  respect  to  reliability  and 
operational  availability;  take  corrective  action  as 
required  and  recognize  and  reward  successes  when  they 
occur . 
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Develop  and  enforce  a  means  to  measure  and  evaluate 
performance  as  it  relates  to  reliability.  This  applies 
throughout  the  life  cycle  inclusive  of  fielded  systems. 
For  systems  in  development  use  of  real-world  test 
conditions  and  large  enough  sample  sizes  for  high 

confidence  statements  is  required. 

C.  CONCLUSIONS  AND  RECOMMENDATIONS 

In  addition  to  answering  the  research  questions,  this 
thesis  has  lead  to  the  following  highlighted  topics:  1) 

Reliability  Management  Culture;  2)  Reliability  Test  and 
Evaluation;  3)  Reliability  Standards.  Each  of  these  topics 
is  presented  with  a  conclusion  and  a  corresponding 
recommendation . 

1 .  Reliability  Management  Culture 

Conclusion : 

Reliability  management  exists  in  an  environment  that 
does  not  recognize  its  value  or  reward  its  successes. 

PMs  are  evaluated  solely  on  cost,  schedule  and 
performance  in  the  production  arena.  Unfortunately, 

reliability  optimization  costs  both  in  terms  of  budget  and 
time,  and  reliability  is  generally  not  regarded  as  a 
performance  item.  Conceptually  the  acquisition  workforce 
recognizes  reliability  as  a  significant  issue  but  it  is 
lost  in  the  crisis  management  atmosphere  that  encompasses 
under  funded  acquisition  management. 

There  is  no  effective  champion  for  reliability  issues. 
Admittedly,  the  Defense  Department  is  not  designed  for 

efficiency  but  rather  works  on  a  system  of  checks  and 
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balances  and  under  funded  budgeting  practices.  However, 
the  acknowledgement  of  reliability  as  a  life  cycle  cost 
driver  warrants  the  implementation  of  proactive  measures. 

Recommendation : 

DoD/DoN  leadership  should  develop  a  performance 
measure  and  reward  structure  that  acknowledges  life  cycle 
support  equal  to  or  higher  to  the  current  cost,  schedule 
and  performance  system. 

Cultural  change  is  required.  A  reliability  and  life 
cycle  focused  education  must  be  incorporated  into  the 
acquisition  workforce  profession.  Specific  reliability 
attention  must  be  given  to  the  PM  and  the  logistician 
profession . 

A  champion  for  reliability  must  emerge.  Consideration 
should  be  given  to  a  flag  level  position  of  a  Reliability 
CZAR. 

Additionally,  the  Defense  Acquisition  Executive 
Summary  (DAES) ,  which  requires  Program  Managers  to  address 
cost,  schedule,  and  performance,  should  also  include 
Operational  Availability  (Aq'  .  The  fourth  dimension  of 
availability  should  mandate  an  Aq  level  identifying  the 
agreed  upon  target. 

2.  Reliability  Test  and  Evaluation 

Conclusion : 

The  relationship  between  program  management  and  test 
and  evaluation  personnel  is  often  adversarial  beyond 
independence . 
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Testing  is  truly  a  value  added  event  that  is  required 
for  effective  reliability  management.  However,  the  current 
bounds  of  the  PPBS  and  the  zero  defect  mentality  that  many 
programs  face  does  not  encourage  testing.  Often  decision 
points  and  funding  sources  are  inflexible  leading  PMs  to 
compress  or  cut  testing  to  inadequate  levels. 

Recommendation : 

DoD/DoN  leadership  should  reevaluate  the  rigidity  that 
exists  in  our  funding  and  decision  cycles  and  encourage  a 
knowledge  gained  approach  to  testing. 


PMs  need  to  embrace  the  lessons  learned  through 
testing  and  allow  the  corrective  actions  to  be  retested  to 
ensure  the  desired  results  have  been  obtained.  If  the 
negative  connotation  of  "failing"  a  DT  test  is  prevalent 
then  we  have  missed  the  entire  learning  point  of  trying  to 
improve  the  system. 


Developmental  testers  should  be  integrated  into  the 
development  team  and  not  relegated  to  messenger  duty  when 
problems  or  issues  arise.  DT  should  be  the  PMs  truth 
finder  concerning  reliability  claims. 


3.  Reliability  Standards 

Conclusion : 

The  discipline  of  reliability  continues  to  depend  on 
the  outdated  cancelled  MIL-STDs . 

The  era  of  dominant  military  standards  ended  with  the 
implementation  of  specification  and  standard  reform. 
Although  these  standards  continue  to  provide  helpful 
information  they  are  no  longer  updated  or  reviewed.  No 
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commercial  standard  has  emerged  to  replace  the 

Government' s . 

Recommendation : 

DoD/DoN  should  work  with  commercial  and  industry 
sources  to  develop  a  comprehensive  standard  encompassing 
reliability  management  practices  from  a  life  cycle  cost 
perspective  that  would  be  interdisciplinary  in  nature  and 
applicable  to  the  PM,  reliability  engineer,  logistician, 
operational  and  developmental  testers  alike. 

D.  AREAS  FOR  FURTHER  STUDY 

1 .  Reliability  Program  Plan 

Research  In  this  area  should  examine  and  analyze 
existing  PMA  Reliability  Program  Plans.  While  many 

programs  have  established  plans,  none  were  provided  to  this 
researcher.  Plans  should  be  reviewed  for  content  and 
degree  of  compliance  In  program  execution. 

2 .  Reliability  Management  in  Other  DoD  Programs 
Future  research  should  analyze  methods  used  In  other 

areas  of  DoD  acquisition.  This  thesis  concentrated  on 
Naval  Aviation  programs  limiting  the  scope  and  focus. 
Similar  analysis  should  be  applied  to  other  acquisition 
areas  In  order  to  Identify  any  stove  piped  lessons  learned. 
Additionally,  comparative  research  can  be  applied  to 
reliability  management  throughout  the  differing  services  or 
acquisition  management  areas  within  a  service. 

3.  Reliability  Management  in  Commercial  Aviation 
Programs 

Future  research  should  examine  reliability  management 
and  practices  used  In  commercial  aviation  program 

management.  Commercial  aviation  not  only  has  experience  In 
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producing  reliable  equipment  but  in  its  in-service 
operation  as  well.  Lessons  learned  should  be  collected  and 
examined  for  possible  applicability  to  DoD  application. 
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APPENDIX  A.  WEAPON  SYSTEM  RELIABILITY  PERFORMANCE  SURVEY 


Directions;  This  survey  is  being  conducted  to  support  research  as  part 
of  a  Naval  Postgraduate  School  Thesis  on  challenges  in  managing  weapon 
system  reliability  performance .  The  results  of  this  thesis  are 
intended  to  directly  benefit  any  PM  that  is,  or  will  be  managing 
complex  programs ,  by  identifying  common  reliability  management  issues 
and  potential  pitfalls ,  why  they  occur,  risk  mitigation  techniques , 
lessons  learned,  and  suggestions  for  improved  methods  for  managing  and 
reducing  the  inherent  risks  associated  with  achieving  stated 
reliability  performance  requirements . 

The  research  is  limited  to  a  cross-section  of  systems  in  various  stages 
of  the  acquisition  process  that  are  managed  within  the  Naval  Air 
Systems  Command.  The  analysis  is  limited  to  an  assessment  of 
reliability  management  and  process  issues,  and  does  not  specifically 
address  commodity  or  technology  driven  reliability  problems . 

Please  answer  the  following  questions  and  email  them  back  NLT  01  Feb 
2002.  A  separate  survey  is  required  to  be  filled  out  for  each 
participating  program. 
e-mail:  glmasiel@nps.navy.mil 

**  Results  will  be  represented  in  aggregate  form,  not  program  specific 


Pro ject/ Program  Management  Office : select  here 
title) 


(click  on  dropdown  list) 
(or  fill  in  appropriate 


Program/ System  Name: select  here 
title) 


(click  on  dropdown  list) 

(or  fill  in  appropriate 


Current  Life  Cycle  Phase: 

Old  5000 

□  Phase  0  (CE  ) 

□  Phase  I  (PDRR  ) 

□  Phase  II  (EMD  specify  prior  to  or  post  TRIP  ) 

□  Phase  III (Specify  if  prior  to  or  post  lOC-if  post  IOC  how  long  has  it 

been  in  the  field?  years) 

New  5000 

Q  MS  A  (specify  CE  or  CAD  ) 

Q  MS  B  (specify  SI  or  SDD  ) 

□  MS  C  (specify  TRIP  or  ERP  ) 

□  Operations  &  Support  (how  long  has  it  been  in  the  field?  years) 

□  Other  or  N/A  (  ) 
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etc...) 


Required  Reliability/Availability :  (specify  reliability 
requirement/measure  in  terms  of  MTBF ,  MTBCMF ,  MTBOMF ,  MTBMA,  Aq, 

I  I  ORD  (state  value  e.g.  300  hrs  MTBF,  95%  Aq) 

□  Contract  (state  value) 

□  Other  (state  value) 

Measured  Reliability /Availability:  (quantify  measured  reliability 
results  consistent  with  measures/units  from  above,  e.g.  300  hrs  MTBF, 
95%  Ao) 

□  DT  results:  Passed?  yQ  n|  | 

□  RQT/RDGT  results:  Passed?  yQ  nD 

□  OT  results:  Passed?  yQ  n|  I 

□  Field  Data  results:  (how  collected:  ) 

I  I  Contractor  claims  ; 

□  Other  results:  Passed?  yQ  n|  | 

(state  type  of  test:  ) 

Has  the  system  experienced  any  major  reliability  test  failures?  (i.e. 
failed  DT  or  lOTE  reliability  performance  requirements)  YesQ  No|  | 
Explain : 


Survey  Respondent's  Billet  (Optional) 


Please  answer  the  following  survey  questions . 
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Survey  Questions:  (please  answer  all  questions.  If  a  question  does  not 
apply  to  your  program  due  to  its  current  acquisition  phase,  please 
answer  based  on  experiences  encountered  in  prior  phases.  Check  all 
boxes  that  apply.  I  have  left  room  after  each  question  for  additional 
commentary  if  you  find  it  necessary) 

1.  How  is  the  system  reliability  program  and  corresponding  management 
approach  to  such  formally  documented  within  your  program?  (check  only 
the  primary  overriding  document) 

□  Reliability  Program  Plan  □  Contract  SOW  □  TEMP  □ 

SAMP ( SingleAcqMgmtPlan 

□  No  formal  reliability  management  plan  □  Other  {explain:  ) 

Additional  comments: 

2.  Who  within  your  organization  is  primarily  responsible  for 
reliability  activities  for  this  particular  program?  (check  only  one) 

□  PM 

□  Project  Leader 

□  Systems  Engineering  Team  Lead 

□  Logistics/Supportability  Team  Lead 

□  Test  Team  Lead 

□  Reliability  I PT (formally  chartered  IPT?  Y  □  nD) 

□  Prime  Contractor 

□  No  one  specifically 

□  Other  (please  explain  ) 

Additional  comments: 

3 .  What  contractual  design  tools  were/are  employed  to  ensure 
reliability  is  "built  in"  early  on  in  the  program?  (check  all  that 
apply) 

□  Physics  of  Failure  (POE)  techniques 

□  Critical  Items  List/Analysis  (i.e.  complex,  state-of-the-art 
technology ,  high  cost,  single  source,  or  single  failure  point 

component) 

□  Identification  of  potential  reliability  problems  (i.e.  known 
reliability  problem  areas) 

□  Software  Reliability  Assessment 

□  Quality  Function  Deployment  (explain:  ) 

□  Parts  Control  Program 

□  FMECA,  FRACAS,  Fault  Tree  Analysis 

□  Other  (describe:  ) 

Additional  comments: 

4 .  Identify  the  types  of  test  activities  that  have  or  will  be  used  to 
determine  compliance  as  part  of  your  system's  reliability  program. 

(check  all  that  apply) 

□  Environmental  Testing 

I  I  Accelerated  Testing  (e.g.  HALT) 

□  Reliability  Development  Growth  Test  (RDGT) 

I  I  Reliability  Qualification/Demonstration  Test  (RQT  or  RDT) 

□  Government  Developmental  Testing 

□  Operational  Testing  (type,  i.e.  LUT/OPTEMPO/ lOT/FOT  ) 

□  Other  (describe:  ) 

Additional  comments: 
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5.  Is  the  amount  of  time  and  funding  allotted  for  reliability  testing 
during  DT  sufficient  for  your  program?  (for  systems  beyond  DT, 
answer  in  terms  of  how  your  program  was  postured  going  into  DT  at  the 
time) 

I  I  Current  schedule  and  available  funds  are  sufficient  (low  risk 

now) 

I  I  Could  use  more  time/$$  to  reduce  risk  (medium/high  risk  now) 

I  I  No  comment 
Additional  comments : 


6.  Does  your  program  incorporate  a  reliability  growth  program? 

□  Yes  {where  is  this  detailed?  ) 

□  No 

□  N/A  (check  this  only  if  system  is  already  fielded  and  there  are  no 
current  plans  for  improving  the  inherent  system  reliability) 

Additional  comments: 


7.  If  your  system  has  already  participated  in  an  OPEVAL,  did  your 
success  in  either  DT  or  RD/GT  (or  other  reliability  testing)  correlate 
with  success  in  lOTE?  (check  all  that  apply) 

□  Yes,  success  in  pre-OPEVAL  reliability  testing  led  to  reliability 
requirements  being  fully  met  in  OPEVAL 

□  Not  completely,  system  did  well  in  pre-OPEVAL  reliability  testing, 
but  had  some  new  problems  during  OPEVAL  that  needed  correcting 

□  Not  at  first,  system  passed  OPEVAL  after  #  attempts  (click  on 
dropdown  list) 

□  N/A,  system  has  not  yet  been  involved  in  an  operational  test 
Additional  comments: 


a.  To  what  level  was  your  system's  ORD  reliability  requirement 
demonstrated  (state  in  terms  of  %  of  ORD  requirement  met) 


During  DT? 
□  100% 

□  >80% 

□  >60% 

□  >40% 

□  >20% 

□  <20% 


During  OT? 
□  100% 

□  >80% 

□  >60% 

□  >40% 

□  >20% 

□  <20% 


8.  Does  (or  did)  your  program  have  specific  OPEVAL  entrance  criteria 
relative  to  reliability? 

I  I  Yes  (provide  details:  ) 


□  No 


Additional  comments : 


96 


9.  Have  the  User,  Tester,  Contractor,  and  PMO  all  agreed  upon  the 
method  (model)  to  be  used  in  reliability  calculations? 

I  I  Yes  (where  is  this  documented,  e.g.  contract,  TEMP,  SEP??  ) 

□  No 

□  Not  sure 
Additional  comments: 


10 .  Is  Reliability  identified  as  a  Key  Performance  Parameter  (KPP)  in 
the  system  Operational  Requirements  Document? 

I  I  Yes 

□  No 

a.  If  not  a  KPP,  for  systems  still  in  development,  where  is 
reliability  ranked  in  terms  of  requirements?  (Relative  priority) 

□  Highest  tier  priority  (Band  A) 

□  Middle  tier  priority  (Band  B) 

□  Lower  tier  (Band  C  or  below) 

Additional  comments: 

11 .  Were  you  as  the  Material  Developer  able  to  influence  incorporation 
of  realistic  reliability  requirements  as  part  of  the  ORD  process?? 

□  No,  OPNAV  N78  developed  requirements  independently 

□  Yes,  input  was  provided  and  included  part  of  IPT  process 

□  Other  (explain:  ) 

Additional  comments: 

12 .  Was  reliability  included  as  a  factor  in  the  source  selection 
process? 

□  Yes  {provide  details  )  Was  it  a  significant  discriminator  ?  yD  N 

□ 

□  No 

Additional  comments: 

a.  How  are  ORD  reliability  requirements  for  your  program  translated 
into  actual  contractual  reliability  requirements?  (base  on  last 

contract  awarded) 

□  ORD  paragraphs  relative  to  reliability  are  restated  in  SOW/Spec  (i.e. 
contract  requirement  is  equal  to  ORD  requirement) 

□  Additional  levels  of  reliability  are  applied  to  the  contract 
{briefly  describe  process) 

□  Comprehensive  reliability  requirements  are  not  adequately  stated  in 
the  contract 

□  Other  {explain;  ) 

13 .  Are  there  incentives  employed  in  the  contract  that  are  specifically 
tied  to  achieving  system  reliability  performance  requirements? 

□  Yes  {describe:  ) 

□  No 

a.  If  yes,  did  these  incentives  achieve  their  desired  effect? 

I  I  Yes 

□  No 

I  I  Too  early  to  tell 
Additional  comments: 
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14 .  Are  you  aware  of  any  specific  DoD  or  Navy  policy/regulation 
regarding  weapon  system  reliability  management? 

□  Yes  {if  yes,  which  do  you  use  to  help  you  manage  reliability?  ) 

□  No 

□  Not  sure 
Additional  comments: 

15 .  What  risk  mitigation  techniques  does  your  program  employ  that 
address  system  reliability  performance  issues? 

Briefly  describe: 

Additional  comments: 


16 .  How  do  you  measure  and  track  reliability  performance  progress  over 
time  in  your  program?  {check  all  that  apply) 

□  By  contractor  projections/analysis 

□  Reliability  growth  tracking  methodology 
I  I  At  major  reviews  (PDR,  CDR,  TRRs,  etc...) 

□  Other  {please  specify:  ) 

Additional  comments: 


17.  In  your  opinion,  has  acquisition  streamlining  (e.g.  performance 
specifications,  use  of  COTS,  etc...)  and/or  the  continued  trend  of 
government  downsizing  contributed  either  directly  or  indirectly  towards 
reliability  shortfalls  experienced  by  programs  in  general? 

□  Yes,  acquisition  streamlining  {provide  details : 

□  Yes,  government  downsizing  {provide  details : 

□  Yes,  both  {provide  details: 

□  No 

□  No  comment 

a.  If  COTS/NDI  components  were/are  utilized  in  the  design  of  your 
system,  did  the  COTS  components  realize  the  reliability  performance 
claims  of  the  OEM? 

I  I  Met 

□  Exceeded 

□  Less  {provide  details,  e.g.  problems  with  integration,  use  in 
mill tary 

environment ,  improper  claims,  etc...  :  ) 

I  I  N/A  {no  COTS/NDI  in  system  design) 

Additional  comments: 

b.  Given  the  realities  of  streamlining  and  downsizing,  do  you 
believe  the  Navy  reliability  community  has  adequately  compensated  with 
alternative  policies,  processes  and  tools? 

I  I  Yes 

□  No 

□  No  comment 

c.  Do  you  have  any  suggestions  for  improvement?  {explain:  ) 

Additional  comments: 
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18.  For  "fielded"  systems  only,  please  answer  the  following: 

a.  Was  or  is  your  program  fielded  in  a  "conditional  materiel 
release"  status  due  in  part  from  failure  to  meet  ORD  RAM  requirements? 

I  I  Yes  (is  CMR  still  in  effect?  Yes  Q  No|  |) 

□  No 

Additional  comments: 

b.  How  is  collection  of  reliability  field  data  performed  to  gather 
failure  and  repair  histories? 

□  Depot  or  CLS  Maintenance  records 

□  Warranty  data  gives  us  this  information 

□  Reliability  data  not  formally  collected 

□  Other  {explain;  ) 


c.  Does  current  field  reliability  data  indicate  your  system  still 
meets  or  exceeds  the  ORD  reliability  requirement? 

I  I  Yes 

□  No 

□  Reliability  data  not  formally  collected 
Additional  comments: 

d.  Has  any  of  the  reliability  failure  data  collected  led  to 
identification  of  O&S  cost  drivers  that  subsequently  led  to  cost 
effective  improvements? 

□  Yes  {If  significant  improvements,  please  expand  upon:  ) 

□  No 

Additional  comments: 

e.  Is  there  a  formal  reliability  improvement  program  for  your 
system? 

□  Yes  {if  yes,  where  documented?  ) 

□  No 

Additional  comments: 

f .  Does  your  system  employ  a  Reliability  Centered  Maintenance 
program? 

□  Yes  {if  yes,  how  is  it  formally  Implemented?  ) 

□  No 

Additional  comments: 

19.  Does  your  program  employ  or  leverage  any  commercial  best  practices 
in  terms  of  reliability  performance  management?  (e.g.  physics  of 

failure ,  predictive  technologies ,  prognostics/life  consumption 
monitoring ,  identification  and  mitigation  of  failure  modes/mechanisms 
(FMECA) ,  accelerated  life  testing ,  growth  testing ,  selection  of 
reliable  parts) 

□  Yes  {identify :  ) 

□  No 

Additional  comments: 


20 .  Rank  order  the  following  reliability  management  problems : 
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(click  on  dropdown  list  for  each) 


Reliability  is  not  a  KPP 

Contractor  not  designing  for  reliability  sufficiently  above 
requirement 

Contractors  not  using  best  commercial  practices 
Not  aggressively  "designing-in"  reliability  upfront 
Poor  reliability  planning  and  growth  planning  (test  too  late) 
Inadequate  policies  and  guidance  (need  updating) 

Insufficient  reliability  testing  to  verify  requirements 
Unrealistic  reliability  requirements  with  inadequate  rationale 
Need  more  qualified  personnel  in  reliability  management 
Not  consistently  improving  reliability  after  fielding 
Other  {fill  In  your  own:  ) 

Additional  comments: 

Please  provide  any  other  comments,  observations,  or  lessons  learned 
that  you  would  like  to  share  here  (use  additional  sheet  if  necessary 
thank  you  for  your  time  and  support  in  filling  out  this  survey. 
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